You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "chandu-1101 (via GitHub)" <gi...@apache.org> on 2023/07/07 14:15:26 UTC
[GitHub] [hudi] chandu-1101 opened a new issue, #9141: [SUPPORT] unable to write dataframe into s3. Getting: HoodieSparkSqlWriter$: UPSERT failed with errors
chandu-1101 opened a new issue, #9141:
URL: https://github.com/apache/hudi/issues/9141
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
I am trying to merge CDC json data into snapshot. For this I first tookthe dataframe from existing parquet and tried to write to s3 in hudi format. I get the below error.
A clear and concise description of the problem.
1. I am running in spark shell with 3 executors ; each with 3GB memory, 1core. For driver: 1core. 1gb memory.
2. Below is the code with the markup where its failing.
**To Reproduce**
```
import org.apache.hudi.QuickstartUtils
import org.apache.hudi.common.model.HoodieAvroPayload
import org.apache.hudi.common.model.WriteOperationType
import org.apache.hudi.config.HoodieWriteConfig
import org.apache.hudi.keygen.constant.KeyGeneratorOptions
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.api.java.function.Function
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.Row
import org.apache.spark.sql.SparkSession
import java.util
import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
import org.apache.spark.sql.SaveMode.Append
import org.apache.spark.sql.SaveMode.Overwrite
val snapshotDf = Application.spark().read.parquet("s3://bucket/snapshots-test/dbdump/_bid_9223370348443853913/")
val cdcSchema = SparkUtils.getSchema("s3://bucket/schemas/dbdump-schema.json")
val cdcDf = Application.spark().read.schema(cdcSchema).json("s3://bucket/inputs/dbdump/")
/* done */
/* merge them */
snapshotDf.registerTempTable("snapshot");
val snapshotDf2 = Application.spark().sql("select * from snapshot where cdc_oid is not null and cdc_oid !='' ")
val snapshotDf3 = snapshotDf2.withColumn("hash", lit(col("cdc_oid").hashCode() %1000) )
snapshotDf3.write.format("hudi").options(QuickstartUtils.getQuickstartWriteConfigs())
.option(HoodieWriteConfig.PRECOMBINE_FIELD_NAME.key(), "timestamp_in_millis")
.option(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "cdc_oid")
.option(KeyGeneratorOptions.PARTITIONPATH_FIELD_NAME.key(), "hash")
.option(TBL_NAME.key(), "GE11")
.mode(Overwrite)
.save("s3://bucket/snapshots-hudi/ge11/snapshot");
```
Steps to reproduce the behavior:
1. run the above program on parquet files of size 10GB; each row of size 6kb
4.
5.
6.
**Expected behavior**
1. The hudi table should have been created from the snapshot parquet files
2. The merge should have happened from CDC , but before this itself things failed
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :hudi-spark3.3-bundle_2.12-0.12.3.jar
* Spark version : 3.3.0
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) : s3
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```
07-07 14:11:10 WARN DAGScheduler: Broadcasting large task binary with size 1033.6 KiB
07-07 14:12:24 ERROR HoodieSparkSqlWriter$: UPSERT failed with errors
org.apache.hudi.exception.HoodieException: Write to Hudi failed
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:148)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1639855549
Hi,
@ad1happy2go I get the same exception again. Below are the 4 variants I tried.
spark shell command: With -jars taking my custom jar. --package referring to hudi.jar
```
spark-shell --driver-memory 1g --executor-memory 4g --executor-cores 1 --driver-cores 1 --conf spark.dynamicAllocation.maxExecutors=2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.caseSensitive=true" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --name ravic --packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1 --jars /home/hadoop/jars2/spark-1.0-SNAPSHOT.jar
```
spark shell command: With -jars taking my custom jar and hudi.jar
```
spark-shell --driver-memory 1g --executor-memory 4g --executor-cores 1 --driver-cores 1 --conf spark.dynamicAllocation.maxExecutors=2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.caseSensitive=true" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --name ravic --jars /home/hadoop/jars2/spark-1.0-SNAPSHOT.jar,/home/hadoop/hudi/hudi-release-0.12.3/packaging/hudi-spark-bundle/target/hudi
-spark3.3-bundle_2.12-0.12.3.jar
```
spark shell command: With -jars taking hudi.jar (no custom jar)
```
spark-shell --driver-memory 1g --executor-memory 4g --executor-cores 1 --driver-cores 1 --conf spark.dynamicAllocation.maxExecutors=2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.caseSensitive=true" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --name ravic --jars /home/hadoop/hudi/hudi-release-0.12.3/packaging/hudi-spark-bundle/target/hudi-spark3.3-bundle_2.12-0.12.3.jar
```
spark shell command: With only --package switch (no custom jar, no hudi jar)
```
spark-shell --driver-memory 1g --executor-memory 4g --executor-cores 1 --driver-cores 1 --conf spark.dynamicAllocation.maxExecutors=2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.caseSensitive=true" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --name ravic --packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1
```
WARNS I get when I run the code (same above code. exactly same.)
```
07-18 08:58:36 ${sys:config.appname} WARN HoodieSparkSqlWriter$: hoodie table at s3://bucket/snapshots-hudi/ge11-drop/snapshot already exists. Deleting existing data & overwriting with new data.
07-18 08:58:38 ${sys:config.appname} WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.int96RebaseModeInWrite' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.int96RebaseModeInWrite' instead.
07-18 08:58:38 ${sys:config.appname} WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.int96RebaseModeInRead' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.int96RebaseModeInRead' instead.
07-18 08:58:38 ${sys:config.appname} WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.datetimeRebaseModeInWrite' instead.
07-18 08:58:38 ${sys:config.appname} WARN SQLConf: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInRead' has been deprecated in Spark v3.2 and may be removed in the future. Use 'spark.sql.parquet.datetimeRebaseModeInRead' instead.
07-18 08:58:39 ${sys:config.appname} WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
07-18 08:58:39 ${sys:config.appname} WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
07-18 08:58:41 ${sys:config.appname} WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
07-18 08:58:41 ${sys:config.appname} WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore hadoop@172.25.26.169
07-18 08:58:43 ${sys:config.appname} WARN HoodieBackedTableMetadata: Metadata table was not found at path s3://bucket/snapshots-hudi/ge11-drop/snapshot/.hoodie/metadata
```
Exception. Again the same. Please note that we have columns with same name but different case and hence the flags enabled in the spark shell command
```
07-18 08:58:43 ${sys:config.appname} WARN HoodieBackedTableMetadata: Metadata table was not found at path s3://p-crm-messaging-v2/snapshots-hudi/ge11-drop/snapshot/.hoodie/metadata
07-18 08:59:03 ${sys:config.appname} ERROR HoodieSparkSqlWriter$: UPSERT failed with errors
org.apache.hudi.exception.HoodieException: Write to Hudi failed
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:153)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
... 49 elided
scala>
```
Kindly let me know if I can do something more to get this working?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] unable to write dataframe into s3. Getting: HoodieSparkSqlWriter$: UPSERT failed with errors
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1628679634
Any update? is the above explanation missing something important? If i modify the above program to the below, the program returns in few seconds (the parquet file in the src is 40GB). When i go to the destination i see some `.hoodi` folders nothing more.
```
snapshotDf3.write.format("hudi")
// .options(getQuickstartWriteConfigs)
.option(DataSourceWriteOptions.OPERATION.key(), DataSourceWriteOptions.BOOTSTRAP_OPERATION_OPT_VAL)
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "cdc_oid")
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "hash")
.option(HoodieTableConfig.BASE_FILE_FORMAT.key(), HoodieFileFormat.PARQUET.name())
.option(HoodieBootstrapConfig.BASE_PATH.key(), "s3://bucket/snapshots-hudi/ge11/snapshot")
// .option(HoodieWriteConfig.KEYGENERATOR_CLASS_NAME.key(), "org.apache.hudi.keygen.NonpartitionedKeyGenerator")
.option(TABLE_NAME, "GE11")
.mode(SaveMode.Overwrite)
.save("s3://bucket/snapshots-hudi/ge11/snapshot");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [BUG] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1643253927
Hi,
Just wanted to know if this bug is planned to be fixed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1628828240
@chandu-1101 The above code snippet only should work. Can you paste the full stack trace please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1635553168
@chandu-1101 In the command I see you using both --packages and --jars and that too with different versions
```spark-shell --driver-memory 1g --executor-memory 4g --executor-cores 1 --driver-cores 1 --conf spark.dynamicAllocation.maxExecutors=2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.caseSensitive=true" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --name ravic --packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1 --jars /home/hadoop/jars2/spark-1.0-SNAPSHOT.jar,/home/hadoop/h
udi/hudi-release-0.12.3/packaging/hudi-spark-bundle/target/hudi-spark3.3-bundle_2.12-0.12.3.jar```
you should remove either spark bundle in --jars or dont give --packages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 closed issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 closed issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
URL: https://github.com/apache/hudi/issues/9141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1641467424
Are you part of community slack. can we connect over there? You can ping me "Aditya Goenka",
This is a very simple case which is used by lot of users, I can connect and check your setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1630735240
Hi,
After trimming down the data files etc the issue is still reproducable.
1. parquet file has complex data structures ; during the bulk insert we are inserting only 4 rows into hudi table
2. same with CDC (schema file below) ; during merge CDC has only 10 rows
3. I am getting error (below) during the merge of CDC data into the hudi table
4. ran on spark on EMR
code
```
val sess = Application.spark();
/* get snapshot df; cdc df */
val snapshotDf = sess.read.parquet("s3://bucket/snapshots-test/ge11-drop/")
val cdcSchema1 = SparkUtils.getSchema("s3://bucket/schemas/ge11-schema.json")
val cdcDf = sess.read.schema(cdcSchema1).json("s3://bucket/inputs-test/ge11-drop/*")
/* done */
/* merge them */
snapshotDf.createOrReplaceTempView("snapshot")
val snapshotDf2 = snapshotDf.limit(4).withColumn("cdc_pk",lit("0"))
snapshotDf2.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option(DataSourceWriteOptions.OPERATION.key(), WriteOperationType.BULK_INSERT.name())
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "_id.oid")
.option(HoodieWriteConfig.TABLE_NAME,"GE11")
.option(HoodieTableConfig.BASE_FILE_FORMAT.key(), HoodieFileFormat.PARQUET.name())
.option(HoodieBootstrapConfig.BASE_PATH.key(), "s3://bucket/snapshots-hudi/ge11-drop/snapshot")
.option("hoodie.datasource.write.table.type","COPY_ON_WRITE")
.mode(SaveMode.Overwrite)
.save("s3://bucket/snapshots-hudi/ge11-drop/snapshot");
cdcDf.createOrReplaceTempView("cdc")
val _cdcDf = sess.sql("select * from cdc where _id.oid is not null and _id.oid !='' limit 10 ")
_cdcDf.createOrReplaceTempView("_cdc");
_cdcDf.write.format("hudi")
.option(HoodieWriteConfig.PRECOMBINE_FIELD_NAME.key(), "cdc_pk")
.option(KeyGeneratorOptions.RECORDKEY_FIELD_NAME.key(), "_id.oid")
.option("hoodie.datasource.write.operation", "upsert")
.option(TBL_NAME.key(), "GE11")
.mode(SaveMode.Append)
.save("s3://bucket/snapshots-hudi/ge11-drop/snapshot"); // <<<<<< ERROR here
/* done */
```
ERROR
```
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2863)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2799)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2798)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2798)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1239)
at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1239)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1239)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3051)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2993)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2982)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1009)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2229)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2250)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2269)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2294)
at org.apache.spark.rdd.RDD.count(RDD.scala:1274)
at org.apache.hudi.HoodieSparkSqlWriter$.commitAndPerformPostOperations(HoodieSparkSqlWriter.scala:721)
at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:350)
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:145)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
... 59 elided
Caused by: org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :0
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:329)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$mapPartitionsAsRDD$a3ab3c4$1(BaseSparkCommitActionExecutor.java:244)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1(JavaRDDLike.scala:102)
at org.apache.spark.api.java.JavaRDDLike.$anonfun$mapPartitionsWithIndex$1$adapted(JavaRDDLike.scala:102)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2(RDD.scala:907)
at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsWithIndex$2$adapted(RDD.scala:907)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.$anonfun$getOrCompute$1(RDD.scala:378)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1525)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1435)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1499)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1322)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:376)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:327)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:365)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:138)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1516)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.hudi.exception.HoodieException: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: operation has failed
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:166)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdateInternal(BaseSparkCommitActionExecutor.java:358)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpdate(BaseSparkCommitActionExecutor.java:349)
at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:322)
... 28 more
Caused by: org.apache.hudi.exception.HoodieException: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: operation has failed
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:161)
at org.apache.hudi.table.action.commit.HoodieMergeHelper.runMerge(HoodieMergeHelper.java:164)
... 31 more
Caused by: java.util.concurrent.ExecutionException: org.apache.hudi.exception.HoodieException: operation has failed
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.execute(BoundedInMemoryExecutor.java:155)
... 32 more
Caused by: org.apache.hudi.exception.HoodieException: operation has failed
at org.apache.hudi.common.util.queue.BoundedInMemoryQueue.throwExceptionIfFailed(BoundedInMemoryQueue.java:248)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueue.readNextRecord(BoundedInMemoryQueue.java:226)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueue.access$100(BoundedInMemoryQueue.java:52)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueue$QueueIterator.hasNext(BoundedInMemoryQueue.java:278)
at org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:36)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:135)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: org.apache.hudi.exception.HoodieException: unable to read next record from parquet file
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:53)
at org.apache.hudi.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:45)
at org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:106)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
... 4 more
Caused by: org.apache.parquet.io.ParquetDecodingException: Can not read value at 1 in block 0 in file s3://p-crm-messaging-v2/snapshots-hudi/ge11-drop/snapshot/370cddda-d0f5-415a-8338-55135a49f3cf-0_0-337-0_20230711122023911.parquet
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:254)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:132)
at org.apache.parquet.hadoop.ParquetReader.read(ParquetReader.java:136)
at org.apache.hudi.common.util.ParquetReaderIterator.hasNext(ParquetReaderIterator.java:48)
... 8 more
Caused by: java.lang.UnsupportedOperationException: org.apache.parquet.avro.AvroConverters$FieldUTF8Converter
at org.apache.parquet.io.api.PrimitiveConverter.addBoolean(PrimitiveConverter.java:77)
at org.apache.parquet.column.impl.ColumnReaderBase$2$5.writeValue(ColumnReaderBase.java:362)
at org.apache.parquet.column.impl.ColumnReaderBase.writeCurrentValueToConverter(ColumnReaderBase.java:440)
at org.apache.parquet.column.impl.ColumnReaderImpl.writeCurrentValueToConverter(ColumnReaderImpl.java:30)
at org.apache.parquet.io.RecordReaderImplementation.read(RecordReaderImplementation.java:406)
at org.apache.parquet.hadoop.InternalParquetRecordReader.nextKeyValue(InternalParquetRecordReader.java:229)
... 11 more
```
Schemas
CDC
```
_cdc
|-- Spotlight_Calculated: string (nullable = true)
|-- Spotlight_IsFortune500: string (nullable = true)
|-- Spotlight_IsIvyLeague: string (nullable = true)
|-- Spotlight_IsManagement: string (nullable = true)
|-- Spotlight_IsMediaAgencies: string (nullable = true)
|-- Spotlight_IsSeniorRole: string (nullable = true)
|-- Spotlight_workedForCompetetiors: string (nullable = true)
|-- Spotlight_workedForTopCompany: string (nullable = true)
|-- _id: struct (nullable = true)
| |-- oid: string (nullable = true)
|-- additionalConfig: struct (nullable = true)
| |-- recruiterOverWriteOptIn: string (nullable = true)
|-- additionalFields: struct (nullable = true)
| |-- achievements: string (nullable = true)
| |-- atsEmail: string (nullable = true)
| |-- atsId: string (nullable = true)
| |-- availability: string (nullable = true)
| |-- faxNo: string (nullable = true)
| |-- hobbies: string (nullable = true)
| |-- objectives: string (nullable = true)
| |-- passportNo: string (nullable = true)
| |-- references: string (nullable = true)
| |-- summary: string (nullable = true)
|-- additionalRequestInfo: struct (nullable = true)
| |-- optInInfo: struct (nullable = true)
| | |-- atsCandidateApplicationSize: integer (nullable = true)
| | |-- contactNumber: string (nullable = true)
| | |-- currentOptInStatus: integer (nullable = true)
| | |-- dbOptInStatus: integer (nullable = true)
| | |-- emailCommunication: string (nullable = true)
| | |-- oldOptinFormDb: boolean (nullable = true)
| | |-- optIn: boolean (nullable = true)
| | |-- optInFromRequest: boolean (nullable = true)
| | |-- overwriteOptIn: boolean (nullable = true)
| | |-- recruiterOverWriteOptIn: boolean (nullable = true)
| | |-- singleSmsOptIn: boolean (nullable = true)
| | |-- smsOptIn: boolean (nullable = true)
| | |-- source: string (nullable = true)
|-- address: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- addressLine: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- isPrimary: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- region: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- type: string (nullable = true)
| | |-- zipCode: string (nullable = true)
|-- addressLogs: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- addressLines: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- createdDate: string (nullable = true)
| | |-- fieldId: string (nullable = true)
| | |-- isDerived: string (nullable = true)
| | |-- latLong: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- locationIp: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- continentCode: string (nullable = true)
| | | |-- continentName: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- countryIsoCode: string (nullable = true)
| | | |-- latitude: string (nullable = true)
| | | |-- longitude: string (nullable = true)
| | | |-- postalCode: string (nullable = true)
| | | |-- registeredCountry: string (nullable = true)
| | | |-- registeredCountryIsoCode: string (nullable = true)
| | | |-- subDivisions: string (nullable = true)
| | | |-- subDivisionsIsoCode: string (nullable = true)
| | | |-- timeZone: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- location: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- residentialType: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- sourceType: string (nullable = true)
| | |-- standardized: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
| | |-- zipCode: string (nullable = true)
|-- alertData: struct (nullable = true)
| |-- all_fields: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- facetDisplayNames: struct (nullable = true)
| | |-- business: string (nullable = true)
| | |-- businessSegment: string (nullable = true)
| | |-- category: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- experienceLevel: string (nullable = true)
| | |-- managementLevel: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- type: string (nullable = true)
| |-- keywords: string (nullable = true)
| |-- selected_fields: struct (nullable = true)
| | |-- business: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- businessSegment: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- category: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- city: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- country: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- experienceLevel: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- isCritical: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- managementLevel: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- state: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- visibilityType: array (nullable = true)
| | | |-- element: string (containsNull = true)
|-- alertUrl: string (nullable = true)
|-- applySources: array (nullable = true)
| |-- element: string (containsNull = true)
|-- applyTransactionId: string (nullable = true)
|-- atsId: string (nullable = true)
|-- attachments: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- attachmentType: string (nullable = true)
| | |-- bucketId: string (nullable = true)
| | |-- createdDate: string (nullable = true)
| | |-- documentStatus: string (nullable = true)
| | |-- documentType: string (nullable = true)
| | |-- entityIds: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- fieldID: string (nullable = true)
| | |-- isCreatedFromCandidates: string (nullable = true)
| | |-- md5sum: string (nullable = true)
| | |-- parsingDate: string (nullable = true)
| | |-- phenomParser: string (nullable = true)
| | |-- recruiterName: string (nullable = true)
| | |-- recruiterUserId: string (nullable = true)
| | |-- relativePath: string (nullable = true)
| | |-- resumeFileName: string (nullable = true)
| | |-- resumeLanguage: string (nullable = true)
| | |-- resumePath: string (nullable = true)
| | |-- resumeText: string (nullable = true)
|-- avgRating: string (nullable = true)
|-- campaignUnsubscibeType: array (nullable = true)
| |-- element: string (containsNull = true)
|-- candidateActivityTimestamp: string (nullable = true)
|-- candidateCreated: string (nullable = true)
|-- candidateOptInRequest: string (nullable = true)
|-- candidateScore: string (nullable = true)
|-- careerLevel: string (nullable = true)
|-- categories: array (nullable = true)
| |-- element: string (containsNull = true)
|-- categoryList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- certifications: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- certificationName: string (nullable = true)
| | |-- date: string (nullable = true)
| | |-- expiryDate: struct (nullable = true)
| | | |-- year: integer (nullable = true)
| | | |-- month: integer (nullable = true)
| | | |-- day: integer (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- institute: string (nullable = true)
| | |-- issuedDate: struct (nullable = true)
| | | |-- day: integer (nullable = true)
| | | |-- month: integer (nullable = true)
| | | |-- year: integer (nullable = true)
| | |-- name: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- source: string (nullable = true)
| | |-- startDate: string (nullable = true)
|-- childRefNum: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- createdDate: string (nullable = true)
| | |-- refNum: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
|-- company: string (nullable = true)
|-- confRequestPriority: double (nullable = true)
|-- conf_request_source: string (nullable = true)
|-- consent: string (nullable = true)
|-- consentDate: string (nullable = true)
|-- contacts: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- countryCode: string (nullable = true)
| | |-- countryCode_org: string (nullable = true)
| | |-- deviceType: string (nullable = true)
| | |-- extension: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- originalNumber: string (nullable = true)
| | |-- phoneNumber: string (nullable = true)
| | |-- phoneNumberWithoutCountryCode: string (nullable = true)
| | |-- phoneNumber_org: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- priority: double (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- status: string (nullable = true)
| | |-- type: string (nullable = true)
|-- contacts_org: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- countryCode: string (nullable = true)
| | |-- countryCode_org: string (nullable = true)
| | |-- deviceType: string (nullable = true)
| | |-- extension: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- originalNumber: string (nullable = true)
| | |-- phoneNumber: string (nullable = true)
| | |-- phoneNumberWithoutCountryCode: string (nullable = true)
| | |-- phoneNumber_org: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- priority: double (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- status: string (nullable = true)
| | |-- type: string (nullable = true)
|-- createKafkaEvent: string (nullable = true)
|-- createdDate: string (nullable = true)
|-- currentCompany: struct (nullable = true)
| |-- city: string (nullable = true)
| |-- company: string (nullable = true)
| |-- companyUrl: string (nullable = true)
| |-- company_org: string (nullable = true)
| |-- country: string (nullable = true)
| |-- designation: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- industry: string (nullable = true)
| |-- jobLocation: string (nullable = true)
| |-- jobSummary: string (nullable = true)
| |-- jobTitle: string (nullable = true)
| |-- jobTitle_org: string (nullable = true)
| |-- original: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- state: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- providedByUser: string (nullable = true)
| |-- role: string (nullable = true)
| |-- size: string (nullable = true)
| |-- source: string (nullable = true)
| |-- standardizedDate: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- title: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- wikipediaUrl: string (nullable = true)
|-- currentCompanyName: string (nullable = true)
|-- currentCompany_org: struct (nullable = true)
| |-- city: string (nullable = true)
| |-- company: string (nullable = true)
| |-- companyUrl: string (nullable = true)
| |-- company_org: string (nullable = true)
| |-- country: string (nullable = true)
| |-- designation: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- industry: string (nullable = true)
| |-- jobLocation: string (nullable = true)
| |-- jobSummary: string (nullable = true)
| |-- jobTitle: string (nullable = true)
| |-- jobTitle_org: string (nullable = true)
| |-- original: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- state: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- providedByUser: string (nullable = true)
| |-- role: string (nullable = true)
| |-- size: string (nullable = true)
| |-- source: string (nullable = true)
| |-- standardizedDate: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- title: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- wikipediaUrl: string (nullable = true)
|-- dateCreated: string (nullable = true)
|-- derivedAtsEventType: string (nullable = true)
|-- designation: string (nullable = true)
|-- disability: string (nullable = true)
|-- diversity: string (nullable = true)
|-- education: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- aggregate: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- degree: string (nullable = true)
| | |-- degree_org: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- fieldOfStudy: string (nullable = true)
| | |-- fieldOfStudy_org: string (nullable = true)
| | |-- isHighEducation: string (nullable = true)
| | |-- major: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- schoolAndUniversity: string (nullable = true)
| | |-- schoolName: string (nullable = true)
| | |-- schoolName_org: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- universityCity: string (nullable = true)
| | |-- universityCountry: string (nullable = true)
| | |-- universityName: string (nullable = true)
| | |-- universityName_org: string (nullable = true)
| | |-- universityState: string (nullable = true)
| | |-- universityUrl: string (nullable = true)
|-- education_org: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- aggregate: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- degree: string (nullable = true)
| | |-- degree_org: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- fieldOfStudy: string (nullable = true)
| | |-- fieldOfStudy_org: string (nullable = true)
| | |-- isHighEducation: string (nullable = true)
| | |-- major: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- schoolAndUniversity: string (nullable = true)
| | |-- schoolName: string (nullable = true)
| | |-- schoolName_org: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- universityCity: string (nullable = true)
| | |-- universityCountry: string (nullable = true)
| | |-- universityName: string (nullable = true)
| | |-- universityName_org: string (nullable = true)
| | |-- universityState: string (nullable = true)
| | |-- universityUrl: string (nullable = true)
|-- emailList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- email: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- invalid: boolean (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- status: string (nullable = true)
| | |-- type: string (nullable = true)
|-- emails: array (nullable = true)
| |-- element: string (containsNull = true)
|-- ethnicity: string (nullable = true)
|-- eventId: string (nullable = true)
|-- eventName: string (nullable = true)
|-- eventOrganizer: string (nullable = true)
|-- experience: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- company: string (nullable = true)
| | |-- companyUrl: string (nullable = true)
| | |-- company_org: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- designation: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- industry: string (nullable = true)
| | |-- isCurrent: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- jobSummary: string (nullable = true)
| | |-- jobTitle: string (nullable = true)
| | |-- jobTitle_org: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- jobLocation: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- providedByUser: string (nullable = true)
| | |-- role: string (nullable = true)
| | |-- size: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- title: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- wikipediaUrl: string (nullable = true)
|-- experienceRange: struct (nullable = true)
| |-- max: integer (nullable = true)
| |-- min: integer (nullable = true)
| |-- range: string (nullable = true)
| |-- totalExperience: double (nullable = true)
|-- experience_org: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- company: string (nullable = true)
| | |-- companyUrl: string (nullable = true)
| | |-- company_org: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- designation: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- industry: string (nullable = true)
| | |-- isCurrent: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- jobSummary: string (nullable = true)
| | |-- jobTitle: string (nullable = true)
| | |-- jobTitle_org: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- jobLocation: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- providedByUser: string (nullable = true)
| | |-- role: string (nullable = true)
| | |-- size: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- title: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- wikipediaUrl: string (nullable = true)
|-- externalSource: string (nullable = true)
|-- facebookUrl: string (nullable = true)
|-- firstName: string (nullable = true)
|-- firstReferrerId: string (nullable = true)
|-- firstSource: array (nullable = true)
| |-- element: string (containsNull = true)
|-- flsaStatus: string (nullable = true)
|-- gender: string (nullable = true)
|-- geolocation: string (nullable = true)
|-- googleUrl: string (nullable = true)
|-- grade: string (nullable = true)
|-- highestEducation: struct (nullable = true)
| |-- aggregate: string (nullable = true)
| |-- city: string (nullable = true)
| |-- country: string (nullable = true)
| |-- degree: string (nullable = true)
| |-- degree_org: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- fieldOfStudy: string (nullable = true)
| |-- fieldOfStudy_org: string (nullable = true)
| |-- major: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- schoolName: string (nullable = true)
| |-- schoolName_org: string (nullable = true)
| |-- source: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- universityCity: string (nullable = true)
| |-- universityCountry: string (nullable = true)
| |-- universityName: string (nullable = true)
| |-- universityName_org: string (nullable = true)
| |-- universityState: string (nullable = true)
| |-- universityUrl: string (nullable = true)
|-- highestEducation_org: struct (nullable = true)
| |-- aggregate: string (nullable = true)
| |-- city: string (nullable = true)
| |-- country: string (nullable = true)
| |-- degree: string (nullable = true)
| |-- degree_org: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- fieldOfStudy: string (nullable = true)
| |-- fieldOfStudy_org: string (nullable = true)
| |-- major: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- schoolName: string (nullable = true)
| |-- schoolName_org: string (nullable = true)
| |-- source: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- universityCity: string (nullable = true)
| |-- universityCountry: string (nullable = true)
| |-- universityName: string (nullable = true)
| |-- universityName_org: string (nullable = true)
| |-- universityState: string (nullable = true)
| |-- universityUrl: string (nullable = true)
|-- hispanicOrLatino: string (nullable = true)
|-- imEmploymentType: array (nullable = true)
| |-- element: string (containsNull = true)
|-- imInterests: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- imLocations: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- imProfileId: string (nullable = true)
|-- imReferralInterests: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- imTestAccount: string (nullable = true)
|-- importedCandidates: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- applicationId: string (nullable = true)
| | |-- confidential: string (nullable = true)
| | |-- dateCreated: string (nullable = true)
| | |-- franchiseeId: string (nullable = true)
| | |-- jobCategory: string (nullable = true)
| | |-- jobId: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- jobSeekerSource: string (nullable = true)
| | |-- jobSeqNo: string (nullable = true)
| | |-- jobTitle: string (nullable = true)
| | |-- locale: string (nullable = true)
| | |-- referrerCrmUserId: string (nullable = true)
| | |-- referrerImProfileId: string (nullable = true)
| | |-- siteType: string (nullable = true)
|-- instagramUrl: string (nullable = true)
|-- interestLocations: string (nullable = true)
|-- internal: boolean (nullable = true)
|-- isFortune500: string (nullable = true)
|-- isIvyLeague: string (nullable = true)
|-- isJoinTenantCommunity: string (nullable = true)
|-- isSeniorRole: string (nullable = true)
|-- jobCode: string (nullable = true)
|-- jobLevelRefId: string (nullable = true)
|-- jobTitle: string (nullable = true)
|-- language: string (nullable = true)
|-- languageKnown: array (nullable = true)
| |-- element: string (containsNull = true)
|-- languagePreference: string (nullable = true)
|-- lastActivityStatus: struct (nullable = true)
| |-- cand_lastUpdateSource: string (nullable = true)
| |-- internalLastUpdated: string (nullable = true)
| |-- priority_education: double (nullable = true)
| |-- priority_experience: double (nullable = true)
| |-- priority_firstName: double (nullable = true)
| |-- priority_language: double (nullable = true)
| |-- priority_lastName: double (nullable = true)
| |-- priority_location: double (nullable = true)
| |-- priority_middleName: double (nullable = true)
|-- lastName: string (nullable = true)
|-- lastOnboardingDate: string (nullable = true)
|-- lastOnboardingTime: string (nullable = true)
|-- lastUpdateTimestamp: string (nullable = true)
|-- latLong: string (nullable = true)
|-- linkedInMD5Sum: string (nullable = true)
|-- linkedInProfileURL: string (nullable = true)
|-- location: string (nullable = true)
|-- location_org: string (nullable = true)
|-- longTermGoal: string (nullable = true)
|-- md5Check: string (nullable = true)
|-- middleName: string (nullable = true)
|-- migrated: string (nullable = true)
|-- mlFieldsList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- data: struct (nullable = true)
| | | |-- ParseType: string (nullable = true)
| | | |-- ParseVersion: string (nullable = true)
| | | |-- ParsingDate: string (nullable = true)
| | | |-- domain_level_2: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- is_intern: string (nullable = true)
| | | |-- ml_address: struct (nullable = true)
| | | | |-- City: string (nullable = true)
| | | | |-- Address: string (nullable = true)
| | | |-- ml_domain: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- ml_experienceRange: struct (nullable = true)
| | | | |-- max: double (nullable = true)
| | | | |-- min: double (nullable = true)
| | | | |-- range: string (nullable = true)
| | | | |-- totalExperience: double (nullable = true)
| | | |-- ml_jobzone: string (nullable = true)
| | | |-- ml_predictedskills: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | |-- md5Sum: string (nullable = true)
|-- newAddress: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- addressLines: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- createdDate: string (nullable = true)
| | |-- fieldId: string (nullable = true)
| | |-- isDerived: string (nullable = true)
| | |-- latLong: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- locationIp: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- continentCode: string (nullable = true)
| | | |-- continentName: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- countryIsoCode: string (nullable = true)
| | | |-- latitude: string (nullable = true)
| | | |-- longitude: string (nullable = true)
| | | |-- postalCode: string (nullable = true)
| | | |-- registeredCountry: string (nullable = true)
| | | |-- registeredCountryIsoCode: string (nullable = true)
| | | |-- subDivisions: string (nullable = true)
| | | |-- subDivisionsIsoCode: string (nullable = true)
| | | |-- timeZone: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- location: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- residentialType: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- sourceType: string (nullable = true)
| | |-- standardized: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
| | |-- zipCode: string (nullable = true)
|-- newEmail: string (nullable = true)
|-- newLocation: struct (nullable = true)
| |-- addressLines: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- city: string (nullable = true)
| |-- country: string (nullable = true)
| |-- createdDate: string (nullable = true)
| |-- fieldId: string (nullable = true)
| |-- isDerived: string (nullable = true)
| |-- latLong: string (nullable = true)
| |-- location: string (nullable = true)
| |-- locationIp: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- continentCode: string (nullable = true)
| | |-- continentName: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- countryIsoCode: string (nullable = true)
| | |-- latitude: string (nullable = true)
| | |-- longitude: string (nullable = true)
| | |-- postalCode: string (nullable = true)
| | |-- registeredCountry: string (nullable = true)
| | |-- registeredCountryIsoCode: string (nullable = true)
| | |-- subDivisions: string (nullable = true)
| | |-- subDivisionsIsoCode: string (nullable = true)
| | |-- timeZone: string (nullable = true)
| |-- original: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- state: string (nullable = true)
| |-- residentialType: string (nullable = true)
| |-- source: string (nullable = true)
| |-- sourceType: string (nullable = true)
| |-- standardized: string (nullable = true)
| |-- standardizedDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- updatedDate: string (nullable = true)
| |-- zipCode: string (nullable = true)
|-- oldCandidateAnonymous: string (nullable = true)
|-- optIn: boolean (nullable = true)
|-- optInSMS: boolean (nullable = true)
|-- optInSingleSMS: boolean (nullable = true)
|-- optInTextCommunication: boolean (nullable = true)
|-- optInWhatsApp: boolean (nullable = true)
|-- optedOutByRecruiter: boolean (nullable = true)
|-- partnerTenant: string (nullable = true)
|-- passportNumber: string (nullable = true)
|-- pictureS3Url: string (nullable = true)
|-- pictureUrl: string (nullable = true)
|-- preferredCommunicationMode: struct (nullable = true)
| |-- email: boolean (nullable = true)
| |-- sms: boolean (nullable = true)
|-- preferredContactMethod: string (nullable = true)
|-- preferredFirstName: string (nullable = true)
|-- preferredLocationList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- latLong: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- latLong: string (nullable = true)
| | | |-- location: string (nullable = true)
| | | |-- sourceList: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | |-- priority: double (nullable = true)
| | |-- sourceList: array (nullable = true)
| | | |-- element: string (containsNull = true)
|-- prefix: string (nullable = true)
|-- previousEmployeeIds: array (nullable = true)
| |-- element: string (containsNull = true)
|-- profileCompletionStage: string (nullable = true)
|-- profileCreatedBy: struct (nullable = true)
| |-- name: string (nullable = true)
| |-- recruiterUserId: string (nullable = true)
|-- profileEnrichmentDate: string (nullable = true)
|-- profileStatus: struct (nullable = true)
| |-- locale: string (nullable = true)
| |-- statusFinal: string (nullable = true)
| |-- statusKey: string (nullable = true)
| |-- statusLanguage: string (nullable = true)
| |-- updatedDate: string (nullable = true)
|-- profileUpdatesByRecruiter: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- name: string (nullable = true)
| | |-- recruiterUserId: string (nullable = true)
|-- profileUrl: string (nullable = true)
|-- pupId: string (nullable = true)
|-- pushedToATSOn: string (nullable = true)
|-- race: string (nullable = true)
|-- recruiterActivityTimestamp: string (nullable = true)
|-- recruiterEmail: string (nullable = true)
|-- recruiterName: string (nullable = true)
|-- recruiterUserId: string (nullable = true)
|-- recruitersUserId: array (nullable = true)
| |-- element: string (containsNull = true)
|-- refNum: string (nullable = true)
|-- referralSource: string (nullable = true)
|-- relocate: string (nullable = true)
|-- relocationDate: string (nullable = true)
|-- resumeUrl: string (nullable = true)
|-- scopeAttr: string (nullable = true)
|-- shortTermGoal: string (nullable = true)
|-- singleSmsOptInStatus: string (nullable = true)
|-- skillList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- latestDate: string (nullable = true)
| | |-- mlSkills: struct (nullable = true)
| | | |-- skillSco: string (nullable = true)
| | | |-- skill_imp: string (nullable = true)
| | | |-- skill_map: string (nullable = true)
| | | |-- skill_standard: string (nullable = true)
| | |-- score: double (nullable = true)
| | |-- skillSource: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- type: string (nullable = true)
| | |-- value: string (nullable = true)
| | |-- value_org: string (nullable = true)
|-- skillList_org: string (nullable = true)
|-- skills: array (nullable = true)
| |-- element: string (containsNull = true)
|-- smsOptInStatus: string (nullable = true)
|-- socialProfile: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- createdDate: string (nullable = true)
| | |-- emailId: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- md5sum: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- profile: string (nullable = true)
| | |-- profileLink: string (nullable = true)
| | |-- type: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
|-- standardizedLatestDate: string (nullable = true)
|-- swoopProfileId: string (nullable = true)
|-- teamId: string (nullable = true)
|-- tenantSpecific_additionalFields: struct (nullable = true)
| |-- AVIURMajor: string (nullable = true)
| |-- AnyGEbusinesspreference: string (nullable = true)
| |-- AreyouinterestedinhearingfromGE: string (nullable = true)
| |-- Areyouopentorelocation: string (nullable = true)
| |-- AvailabilityDate: string (nullable = true)
| |-- Available: string (nullable = true)
| |-- AviationAreasofInterest: string (nullable = true)
| |-- BFI_DATA: string (nullable = true)
| |-- BusinessOfInterest: string (nullable = true)
| |-- BusinessSegment: string (nullable = true)
| |-- COSDate: string (nullable = true)
| |-- CandidateType: string (nullable = true)
| |-- CareerLevelInterest: string (nullable = true)
| |-- CollegeorUniversity: string (nullable = true)
| |-- Comments: string (nullable = true)
| |-- Consent_Data: string (nullable = true)
| |-- Consent_ExternalTalentShare: string (nullable = true)
| |-- Consent_GDPR: string (nullable = true)
| |-- ContactMeNow_Jobs_Boomerang: string (nullable = true)
| |-- CountriesDesiredtoWork_GECorp: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- CurrentActivity: string (nullable = true)
| |-- CurrentMatric: string (nullable = true)
| |-- DHEvalBusPref: string (nullable = true)
| |-- DareToLead: string (nullable = true)
| |-- DateLeftGE: string (nullable = true)
| |-- DegreeLevel: string (nullable = true)
| |-- Description: string (nullable = true)
| |-- DesiredCompensationRange: string (nullable = true)
| |-- Expected_Graduation_Date: string (nullable = true)
| |-- FN_group_post: string (nullable = true)
| |-- FN_private_editor: string (nullable = true)
| |-- FieldOfStudy: string (nullable = true)
| |-- FormerGEHRManager: string (nullable = true)
| |-- FormerHRM: string (nullable = true)
| |-- FunctionList: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- FunctionsOfInterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- GEAlumni: string (nullable = true)
| |-- GECareerBand: string (nullable = true)
| |-- GEHiredBusiness: string (nullable = true)
| |-- GEHiredBusinessSegment: string (nullable = true)
| |-- GEHiredFunction: string (nullable = true)
| |-- GEHiredJobFamilySelect: string (nullable = true)
| |-- GEHiredRegion: string (nullable = true)
| |-- GEMgmtLevel: string (nullable = true)
| |-- GEPowerInternLocInterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- GPA: string (nullable = true)
| |-- GPAOVERALLDROPDOWN: string (nullable = true)
| |-- GPA_TEXT: string (nullable = true)
| |-- GradMonth: string (nullable = true)
| |-- GradYear: string (nullable = true)
| |-- GraduationMonth: string (nullable = true)
| |-- GraduationYear: string (nullable = true)
| |-- HaveyouheardofGEbefore: string (nullable = true)
| |-- HiredBusSegment: string (nullable = true)
| |-- HowDidYouHeardAboutThisOpportunity: string (nullable = true)
| |-- HowDidYourHearAboutEvent: string (nullable = true)
| |-- HowWouldYouDescribeYourCareerLevel: string (nullable = true)
| |-- HowWouldYouDescribeYourCurrentCareerLevel: string (nullable = true)
| |-- IConsent: string (nullable = true)
| |-- IWantGEToStayConnectedToMeBy: string (nullable = true)
| |-- IWantGEToTellMeAbout: string (nullable = true)
| |-- IWantToHearFromGE: string (nullable = true)
| |-- IfYouHavePriorGEWorkExperiencePleaseExplainYourRole: string (nullable = true)
| |-- Ifyouareateacherregisteringyourstudents: string (nullable = true)
| |-- InWhichCountriesAreYouCurrentlyAuthorizedToWork: string (nullable = true)
| |-- In_Matric_School: string (nullable = true)
| |-- Interest: string (nullable = true)
| |-- InterestUpdate: string (nullable = true)
| |-- InternalBusinessPreference: string (nullable = true)
| |-- InterviewDate: string (nullable = true)
| |-- Interviewavailabilities: string (nullable = true)
| |-- IsGPARankOnA4Scale: string (nullable = true)
| |-- LMWindCanadaSelectJob: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- LMWindCheckbox: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- LanguagesSpoken: string (nullable = true)
| |-- LastEmailedDate: string (nullable = true)
| |-- LeadershipProgramInterest: string (nullable = true)
| |-- Major: string (nullable = true)
| |-- MajorOrAreaOfStudy: string (nullable = true)
| |-- MalaysiaorSingapore: string (nullable = true)
| |-- MatricCompYear: string (nullable = true)
| |-- Matric_Subjects: string (nullable = true)
| |-- Name_Person_Submitting_on_Behalf_Of_Employee: string (nullable = true)
| |-- NameofSchoolCurrentlyAttended: string (nullable = true)
| |-- NewHireRole: string (nullable = true)
| |-- OpenToRelocation: string (nullable = true)
| |-- OperationallyReadyDate: string (nullable = true)
| |-- OtherCollegeorUniversity: string (nullable = true)
| |-- OtherLanguage: string (nullable = true)
| |-- OverallGPA: string (nullable = true)
| |-- PhDCompleted: string (nullable = true)
| |-- PrefPlace_Business: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- PrefPlace_HighestEducation: string (nullable = true)
| |-- PrefPlace_JobDuties: string (nullable = true)
| |-- PrefPlace_Location: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- QualRegister: string (nullable = true)
| |-- ReadinesstoMove: string (nullable = true)
| |-- Relocation_Employee_Talent_Community: string (nullable = true)
| |-- RelotoWhere: string (nullable = true)
| |-- Richmond_Areas_of_Interest: string (nullable = true)
| |-- Richmond_Certificate_Pursued: string (nullable = true)
| |-- Richmond_Shifts_of_Interest: string (nullable = true)
| |-- SG_Education: string (nullable = true)
| |-- SLG_lng_from: string (nullable = true)
| |-- SLG_lng_to: string (nullable = true)
| |-- SL_lng_from: string (nullable = true)
| |-- SL_lng_to: string (nullable = true)
| |-- SSO_ID: string (nullable = true)
| |-- SalaryExpectations: string (nullable = true)
| |-- Schenectady_Jobs_of_Interest: string (nullable = true)
| |-- SchoolEmail: string (nullable = true)
| |-- SecurityClearance: string (nullable = true)
| |-- Shifts_of_Interest: string (nullable = true)
| |-- SkillBridgeAvailDate: string (nullable = true)
| |-- SkillBridgeCmdAck: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- SkillBridgeGeo: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- SkillBridgeNotes: string (nullable = true)
| |-- TargetedAreasofInterest: string (nullable = true)
| |-- Testfield: string (nullable = true)
| |-- TypeofDegree: string (nullable = true)
| |-- UCAlumni: string (nullable = true)
| |-- UCStatus: string (nullable = true)
| |-- URBusinessPreference: string (nullable = true)
| |-- URGradYear: string (nullable = true)
| |-- URMajor: string (nullable = true)
| |-- URPositionSeeking: string (nullable = true)
| |-- UR_Job_Type_Seeking: string (nullable = true)
| |-- URfunctionofinterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- URroletypeofinterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- USExportControl: string (nullable = true)
| |-- USMilAoI: string (nullable = true)
| |-- USMilEASDate: string (nullable = true)
| |-- USMilExperience: string (nullable = true)
| |-- USMilRank: string (nullable = true)
| |-- USMilitaryBranchofService: string (nullable = true)
| |-- USVisaSponsorship: string (nullable = true)
| |-- US_Sponsorship_Needed: string (nullable = true)
| |-- UnderOrPostGradReg: string (nullable = true)
| |-- UniversityOpenText: string (nullable = true)
| |-- University_ModuleSA: string (nullable = true)
| |-- WHATISYOURLEVELOFPARTICIPATION: string (nullable = true)
| |-- Whichfuturebusinessunitsinterestyou: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- YEARINSCHOOL: string (nullable = true)
| |-- YelloRefNumber: string (nullable = true)
| |-- YrsOfExp: string (nullable = true)
| |-- candidatesSMSOptIn: boolean (nullable = true)
| |-- candidatesSingleSMSOptIn: boolean (nullable = true)
| |-- co_speechnotes_plugin_fab_language_selector_id: string (nullable = true)
| |-- currentCollege: string (nullable = true)
| |-- currentOverallGpa: string (nullable = true)
| |-- degree: string (nullable = true)
| |-- degreeLevel: string (nullable = true)
| |-- education: string (nullable = true)
| |-- educationEndDate: string (nullable = true)
| |-- employee_flag: string (nullable = true)
| |-- employee_number: string (nullable = true)
| |-- employee_referral: string (nullable = true)
| |-- enrichmentName: string (nullable = true)
| |-- feedBackInfo: string (nullable = true)
| |-- fieldOfStudy: string (nullable = true)
| |-- field_of_study_italy_Events: string (nullable = true)
| |-- field_of_study_text: string (nullable = true)
| |-- formId: string (nullable = true)
| |-- formType: string (nullable = true)
| |-- gradMonth: string (nullable = true)
| |-- gradYear: string (nullable = true)
| |-- gradyearnew: string (nullable = true)
| |-- hiredBusiness: string (nullable = true)
| |-- hiredBusinessSegment: string (nullable = true)
| |-- initiativesofInterest: string (nullable = true)
| |-- initiativesofInterest_5: string (nullable = true)
| |-- major: string (nullable = true)
| |-- operatingPartners: string (nullable = true)
| |-- optinsinglesms: boolean (nullable = true)
| |-- optinsms: boolean (nullable = true)
| |-- otherCollegeorUniversity: string (nullable = true)
| |-- schoolName: string (nullable = true)
| |-- select_language: string (nullable = true)
| |-- skillSet: string (nullable = true)
| |-- sourcerName: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- testField: string (nullable = true)
| |-- universityName: string (nullable = true)
| |-- userId: string (nullable = true)
| |-- verifyReg: string (nullable = true)
|-- timeZone: string (nullable = true)
|-- topRatedSkills: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- trafficSourceList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- createdDate: string (nullable = true)
| | |-- id: string (nullable = true)
| | |-- rawUrl: string (nullable = true)
| | |-- referralType: string (nullable = true)
| | |-- sourceFullname: string (nullable = true)
| | |-- sourceShortName: string (nullable = true)
| | |-- utmCampaign: string (nullable = true)
| | |-- utmMedium: string (nullable = true)
|-- twitterUrl: string (nullable = true)
|-- updatedDate: string (nullable = true)
|-- userId: string (nullable = true)
|-- userStatus: string (nullable = true)
|-- userType: string (nullable = true)
|-- utids: array (nullable = true)
| |-- element: string (containsNull = true)
|-- veteran: string (nullable = true)
|-- workAuthorization: string (nullable = true)
|-- zipCode: string (nullable = true)
|-- cdc_modifiedon: string (nullable = true)
|-- cdc_pk: string (nullable = true)
|-- cdc_operation: string (nullable = true)
|-- cdc_oid: string (nullable = true)
|-- updatedOn: string (nullable = true)
```
Snapshot dataframe 2 schema
```
snapshotDf2
|-- cdc_pk: string (nullable = false)
|-- cdc_oid: string (nullable = true)
|-- Spotlight_Calculated: boolean (nullable = true)
|-- Spotlight_IsFortune500: boolean (nullable = true)
|-- Spotlight_IsIvyLeague: boolean (nullable = true)
|-- Spotlight_IsManagement: boolean (nullable = true)
|-- Spotlight_IsMediaAgencies: boolean (nullable = true)
|-- Spotlight_IsSeniorRole: boolean (nullable = true)
|-- Spotlight_workedForCompetetiors: boolean (nullable = true)
|-- Spotlight_workedForTopCompany: boolean (nullable = true)
|-- _id: struct (nullable = true)
| |-- oid: string (nullable = true)
|-- additionalConfig: struct (nullable = true)
| |-- recruiterOverWriteOptIn: boolean (nullable = true)
|-- additionalFields: struct (nullable = true)
| |-- achievements: string (nullable = true)
| |-- atsEmail: string (nullable = true)
| |-- atsId: string (nullable = true)
| |-- availability: string (nullable = true)
| |-- faxNo: string (nullable = true)
| |-- hobbies: string (nullable = true)
| |-- objectives: string (nullable = true)
| |-- passportNo: string (nullable = true)
| |-- references: string (nullable = true)
| |-- summary: string (nullable = true)
|-- additionalRequestInfo: struct (nullable = true)
| |-- optInInfo: struct (nullable = true)
| | |-- atsCandidateApplicationSize: integer (nullable = true)
| | |-- contactNumber: string (nullable = true)
| | |-- currentOptInStatus: integer (nullable = true)
| | |-- dbOptInStatus: integer (nullable = true)
| | |-- emailCommunication: string (nullable = true)
| | |-- oldOptinFormDb: boolean (nullable = true)
| | |-- optIn: boolean (nullable = true)
| | |-- optInFromRequest: boolean (nullable = true)
| | |-- overwriteOptIn: boolean (nullable = true)
| | |-- recruiterOverWriteOptIn: boolean (nullable = true)
| | |-- singleSmsOptIn: boolean (nullable = true)
| | |-- smsOptIn: boolean (nullable = true)
| | |-- source: string (nullable = true)
|-- address: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- addressLine: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- isPrimary: boolean (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- region: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- type: string (nullable = true)
| | |-- zipCode: string (nullable = true)
|-- addressLogs: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- addressLines: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- createdDate: string (nullable = true)
| | |-- fieldId: string (nullable = true)
| | |-- isDerived: boolean (nullable = true)
| | |-- latLong: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- locationIp: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- continentCode: string (nullable = true)
| | | |-- continentName: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- countryIsoCode: string (nullable = true)
| | | |-- latitude: string (nullable = true)
| | | |-- longitude: string (nullable = true)
| | | |-- postalCode: string (nullable = true)
| | | |-- registeredCountry: string (nullable = true)
| | | |-- registeredCountryIsoCode: string (nullable = true)
| | | |-- subDivisions: string (nullable = true)
| | | |-- subDivisionsIsoCode: string (nullable = true)
| | | |-- timeZone: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- location: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- residentialType: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- sourceType: string (nullable = true)
| | |-- standardized: boolean (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
| | |-- zipCode: string (nullable = true)
|-- alertData: struct (nullable = true)
| |-- all_fields: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- facetDisplayNames: struct (nullable = true)
| | |-- business: string (nullable = true)
| | |-- businessSegment: string (nullable = true)
| | |-- category: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- experienceLevel: string (nullable = true)
| | |-- managementLevel: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- type: string (nullable = true)
| |-- keywords: string (nullable = true)
| |-- selected_fields: struct (nullable = true)
| | |-- business: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- businessSegment: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- category: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- city: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- country: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- experienceLevel: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- isCritical: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- managementLevel: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- state: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- visibilityType: array (nullable = true)
| | | |-- element: string (containsNull = true)
|-- alertUrl: string (nullable = true)
|-- applySources: array (nullable = true)
| |-- element: string (containsNull = true)
|-- applyTransactionId: string (nullable = true)
|-- atsId: string (nullable = true)
|-- attachments: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- attachmentType: string (nullable = true)
| | |-- bucketId: string (nullable = true)
| | |-- createdDate: string (nullable = true)
| | |-- documentStatus: string (nullable = true)
| | |-- documentType: string (nullable = true)
| | |-- entityIds: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- fieldID: string (nullable = true)
| | |-- isCreatedFromCandidates: boolean (nullable = true)
| | |-- md5sum: string (nullable = true)
| | |-- parsingDate: string (nullable = true)
| | |-- phenomParser: string (nullable = true)
| | |-- recruiterName: string (nullable = true)
| | |-- recruiterUserId: string (nullable = true)
| | |-- relativePath: string (nullable = true)
| | |-- resumeFileName: string (nullable = true)
| | |-- resumeLanguage: string (nullable = true)
| | |-- resumePath: string (nullable = true)
| | |-- resumeText: string (nullable = true)
|-- avgRating: string (nullable = true)
|-- campaignUnsubscibeType: array (nullable = true)
| |-- element: string (containsNull = true)
|-- candidateActivityTimestamp: string (nullable = true)
|-- candidateCreated: string (nullable = true)
|-- candidateOptInRequest: boolean (nullable = true)
|-- candidateScore: string (nullable = true)
|-- careerLevel: string (nullable = true)
|-- categories: array (nullable = true)
| |-- element: string (containsNull = true)
|-- categoryList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- certifications: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- certificationName: string (nullable = true)
| | |-- date: string (nullable = true)
| | |-- expiryDate: struct (nullable = true)
| | | |-- year: integer (nullable = true)
| | | |-- month: integer (nullable = true)
| | | |-- day: integer (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- institute: string (nullable = true)
| | |-- issuedDate: struct (nullable = true)
| | | |-- day: integer (nullable = true)
| | | |-- month: integer (nullable = true)
| | | |-- year: integer (nullable = true)
| | |-- name: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- source: string (nullable = true)
| | |-- startDate: string (nullable = true)
|-- childRefNum: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- createdDate: string (nullable = true)
| | |-- refNum: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
|-- company: string (nullable = true)
|-- confRequestPriority: double (nullable = true)
|-- conf_request_source: string (nullable = true)
|-- consent: string (nullable = true)
|-- consentDate: string (nullable = true)
|-- contacts: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- countryCode: string (nullable = true)
| | |-- countryCode_org: string (nullable = true)
| | |-- deviceType: string (nullable = true)
| | |-- extension: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- originalNumber: string (nullable = true)
| | |-- phoneNumber: string (nullable = true)
| | |-- phoneNumberWithoutCountryCode: string (nullable = true)
| | |-- phoneNumber_org: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- priority: double (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- status: string (nullable = true)
| | |-- type: string (nullable = true)
|-- contacts_org: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- countryCode: string (nullable = true)
| | |-- countryCode_org: string (nullable = true)
| | |-- deviceType: string (nullable = true)
| | |-- extension: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- originalNumber: string (nullable = true)
| | |-- phoneNumber: string (nullable = true)
| | |-- phoneNumberWithoutCountryCode: string (nullable = true)
| | |-- phoneNumber_org: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- priority: double (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- status: string (nullable = true)
| | |-- type: string (nullable = true)
|-- createKafkaEvent: boolean (nullable = true)
|-- createdDate: string (nullable = true)
|-- currentCompany: struct (nullable = true)
| |-- city: string (nullable = true)
| |-- company: string (nullable = true)
| |-- companyUrl: string (nullable = true)
| |-- company_org: string (nullable = true)
| |-- country: string (nullable = true)
| |-- designation: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- industry: string (nullable = true)
| |-- jobLocation: string (nullable = true)
| |-- jobSummary: string (nullable = true)
| |-- jobTitle: string (nullable = true)
| |-- jobTitle_org: string (nullable = true)
| |-- original: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- state: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- providedByUser: boolean (nullable = true)
| |-- role: string (nullable = true)
| |-- size: string (nullable = true)
| |-- source: string (nullable = true)
| |-- standardizedDate: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- title: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- wikipediaUrl: string (nullable = true)
|-- currentCompanyName: string (nullable = true)
|-- currentCompany_org: struct (nullable = true)
| |-- city: string (nullable = true)
| |-- company: string (nullable = true)
| |-- companyUrl: string (nullable = true)
| |-- company_org: string (nullable = true)
| |-- country: string (nullable = true)
| |-- designation: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- industry: string (nullable = true)
| |-- jobLocation: string (nullable = true)
| |-- jobSummary: string (nullable = true)
| |-- jobTitle: string (nullable = true)
| |-- jobTitle_org: string (nullable = true)
| |-- original: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- state: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- providedByUser: boolean (nullable = true)
| |-- role: string (nullable = true)
| |-- size: string (nullable = true)
| |-- source: string (nullable = true)
| |-- standardizedDate: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- title: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- wikipediaUrl: string (nullable = true)
|-- dateCreated: string (nullable = true)
|-- derivedAtsEventType: string (nullable = true)
|-- designation: string (nullable = true)
|-- disability: string (nullable = true)
|-- diversity: string (nullable = true)
|-- education: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- aggregate: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- degree: string (nullable = true)
| | |-- degree_org: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- fieldOfStudy: string (nullable = true)
| | |-- fieldOfStudy_org: string (nullable = true)
| | |-- isHighEducation: boolean (nullable = true)
| | |-- major: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- schoolAndUniversity: string (nullable = true)
| | |-- schoolName: string (nullable = true)
| | |-- schoolName_org: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- universityCity: string (nullable = true)
| | |-- universityCountry: string (nullable = true)
| | |-- universityName: string (nullable = true)
| | |-- universityName_org: string (nullable = true)
| | |-- universityState: string (nullable = true)
| | |-- universityUrl: string (nullable = true)
|-- education_org: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- aggregate: string (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- degree: string (nullable = true)
| | |-- degree_org: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- fieldOfStudy: string (nullable = true)
| | |-- fieldOfStudy_org: string (nullable = true)
| | |-- isHighEducation: boolean (nullable = true)
| | |-- major: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- schoolAndUniversity: string (nullable = true)
| | |-- schoolName: string (nullable = true)
| | |-- schoolName_org: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- universityCity: string (nullable = true)
| | |-- universityCountry: string (nullable = true)
| | |-- universityName: string (nullable = true)
| | |-- universityName_org: string (nullable = true)
| | |-- universityState: string (nullable = true)
| | |-- universityUrl: string (nullable = true)
|-- emailList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- email: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- invalid: boolean (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- status: string (nullable = true)
| | |-- type: string (nullable = true)
|-- emails: array (nullable = true)
| |-- element: string (containsNull = true)
|-- ethnicity: string (nullable = true)
|-- eventId: string (nullable = true)
|-- eventName: string (nullable = true)
|-- eventOrganizer: boolean (nullable = true)
|-- experience: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- company: string (nullable = true)
| | |-- companyUrl: string (nullable = true)
| | |-- company_org: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- designation: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- industry: string (nullable = true)
| | |-- isCurrent: boolean (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- jobSummary: string (nullable = true)
| | |-- jobTitle: string (nullable = true)
| | |-- jobTitle_org: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- jobLocation: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- providedByUser: boolean (nullable = true)
| | |-- role: string (nullable = true)
| | |-- size: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- title: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- wikipediaUrl: string (nullable = true)
|-- experienceRange: struct (nullable = true)
| |-- max: integer (nullable = true)
| |-- min: integer (nullable = true)
| |-- range: string (nullable = true)
| |-- totalExperience: double (nullable = true)
|-- experience_org: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- company: string (nullable = true)
| | |-- companyUrl: string (nullable = true)
| | |-- company_org: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- designation: string (nullable = true)
| | |-- endDate: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- industry: string (nullable = true)
| | |-- isCurrent: boolean (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- jobSummary: string (nullable = true)
| | |-- jobTitle: string (nullable = true)
| | |-- jobTitle_org: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- jobLocation: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- providedByUser: boolean (nullable = true)
| | |-- role: string (nullable = true)
| | |-- size: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- startDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- title: string (nullable = true)
| | |-- transactionId: string (nullable = true)
| | |-- wikipediaUrl: string (nullable = true)
|-- externalSource: string (nullable = true)
|-- facebookUrl: string (nullable = true)
|-- firstName: string (nullable = true)
|-- firstReferrerId: string (nullable = true)
|-- firstSource: array (nullable = true)
| |-- element: string (containsNull = true)
|-- flsaStatus: string (nullable = true)
|-- gender: string (nullable = true)
|-- geolocation: string (nullable = true)
|-- googleUrl: string (nullable = true)
|-- grade: string (nullable = true)
|-- highestEducation: struct (nullable = true)
| |-- aggregate: string (nullable = true)
| |-- city: string (nullable = true)
| |-- country: string (nullable = true)
| |-- degree: string (nullable = true)
| |-- degree_org: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- fieldOfStudy: string (nullable = true)
| |-- fieldOfStudy_org: string (nullable = true)
| |-- major: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- schoolName: string (nullable = true)
| |-- schoolName_org: string (nullable = true)
| |-- source: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- universityCity: string (nullable = true)
| |-- universityCountry: string (nullable = true)
| |-- universityName: string (nullable = true)
| |-- universityName_org: string (nullable = true)
| |-- universityState: string (nullable = true)
| |-- universityUrl: string (nullable = true)
|-- highestEducation_org: struct (nullable = true)
| |-- aggregate: string (nullable = true)
| |-- city: string (nullable = true)
| |-- country: string (nullable = true)
| |-- degree: string (nullable = true)
| |-- degree_org: string (nullable = true)
| |-- endDate: string (nullable = true)
| |-- fieldID: string (nullable = true)
| |-- fieldOfStudy: string (nullable = true)
| |-- fieldOfStudy_org: string (nullable = true)
| |-- major: string (nullable = true)
| |-- priority: double (nullable = true)
| |-- schoolName: string (nullable = true)
| |-- schoolName_org: string (nullable = true)
| |-- source: string (nullable = true)
| |-- startDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- transactionId: string (nullable = true)
| |-- universityCity: string (nullable = true)
| |-- universityCountry: string (nullable = true)
| |-- universityName: string (nullable = true)
| |-- universityName_org: string (nullable = true)
| |-- universityState: string (nullable = true)
| |-- universityUrl: string (nullable = true)
|-- hispanicOrLatino: string (nullable = true)
|-- imEmploymentType: array (nullable = true)
| |-- element: string (containsNull = true)
|-- imInterests: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- imLocations: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- imProfileId: string (nullable = true)
|-- imReferralInterests: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- imTestAccount: boolean (nullable = true)
|-- importedCandidates: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- applicationId: string (nullable = true)
| | |-- confidential: boolean (nullable = true)
| | |-- dateCreated: string (nullable = true)
| | |-- franchiseeId: string (nullable = true)
| | |-- jobCategory: string (nullable = true)
| | |-- jobId: string (nullable = true)
| | |-- jobLocation: string (nullable = true)
| | |-- jobSeekerSource: string (nullable = true)
| | |-- jobSeqNo: string (nullable = true)
| | |-- jobTitle: string (nullable = true)
| | |-- locale: string (nullable = true)
| | |-- referrerCrmUserId: string (nullable = true)
| | |-- referrerImProfileId: string (nullable = true)
| | |-- siteType: string (nullable = true)
|-- instagramUrl: string (nullable = true)
|-- interestLocations: string (nullable = true)
|-- internal: boolean (nullable = true)
|-- isFortune500: string (nullable = true)
|-- isIvyLeague: string (nullable = true)
|-- isJoinTenantCommunity: boolean (nullable = true)
|-- isSeniorRole: string (nullable = true)
|-- jobCode: string (nullable = true)
|-- jobLevelRefId: string (nullable = true)
|-- jobTitle: string (nullable = true)
|-- language: string (nullable = true)
|-- languageKnown: array (nullable = true)
| |-- element: string (containsNull = true)
|-- languagePreference: string (nullable = true)
|-- lastActivityStatus: struct (nullable = true)
| |-- cand_lastUpdateSource: string (nullable = true)
| |-- internalLastUpdated: string (nullable = true)
| |-- priority_education: double (nullable = true)
| |-- priority_experience: double (nullable = true)
| |-- priority_firstName: double (nullable = true)
| |-- priority_language: double (nullable = true)
| |-- priority_lastName: double (nullable = true)
| |-- priority_location: double (nullable = true)
| |-- priority_middleName: double (nullable = true)
|-- lastName: string (nullable = true)
|-- lastOnboardingDate: string (nullable = true)
|-- lastOnboardingTime: string (nullable = true)
|-- lastUpdateTimestamp: string (nullable = true)
|-- latLong: string (nullable = true)
|-- linkedInMD5Sum: string (nullable = true)
|-- linkedInProfileURL: string (nullable = true)
|-- location: string (nullable = true)
|-- location_org: string (nullable = true)
|-- longTermGoal: string (nullable = true)
|-- md5Check: string (nullable = true)
|-- middleName: string (nullable = true)
|-- migrated: boolean (nullable = true)
|-- mlFieldsList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- data: struct (nullable = true)
| | | |-- ParseType: string (nullable = true)
| | | |-- ParseVersion: string (nullable = true)
| | | |-- ParsingDate: string (nullable = true)
| | | |-- domain_level_2: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- is_intern: boolean (nullable = true)
| | | |-- ml_address: struct (nullable = true)
| | | | |-- City: string (nullable = true)
| | | | |-- Address: string (nullable = true)
| | | |-- ml_domain: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | | |-- ml_experienceRange: struct (nullable = true)
| | | | |-- max: double (nullable = true)
| | | | |-- min: double (nullable = true)
| | | | |-- range: string (nullable = true)
| | | | |-- totalExperience: double (nullable = true)
| | | |-- ml_jobzone: string (nullable = true)
| | | |-- ml_predictedskills: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | |-- md5Sum: string (nullable = true)
|-- newAddress: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- addressLines: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- createdDate: string (nullable = true)
| | |-- fieldId: string (nullable = true)
| | |-- isDerived: boolean (nullable = true)
| | |-- latLong: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- locationIp: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- continentCode: string (nullable = true)
| | | |-- continentName: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- countryIsoCode: string (nullable = true)
| | | |-- latitude: string (nullable = true)
| | | |-- longitude: string (nullable = true)
| | | |-- postalCode: string (nullable = true)
| | | |-- registeredCountry: string (nullable = true)
| | | |-- registeredCountryIsoCode: string (nullable = true)
| | | |-- subDivisions: string (nullable = true)
| | | |-- subDivisionsIsoCode: string (nullable = true)
| | | |-- timeZone: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- location: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- residentialType: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- sourceType: string (nullable = true)
| | |-- standardized: boolean (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
| | |-- zipCode: string (nullable = true)
|-- newEmail: boolean (nullable = true)
|-- newLocation: struct (nullable = true)
| |-- addressLines: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- city: string (nullable = true)
| |-- country: string (nullable = true)
| |-- createdDate: string (nullable = true)
| |-- fieldId: string (nullable = true)
| |-- isDerived: boolean (nullable = true)
| |-- latLong: string (nullable = true)
| |-- location: string (nullable = true)
| |-- locationIp: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- continentCode: string (nullable = true)
| | |-- continentName: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- countryIsoCode: string (nullable = true)
| | |-- latitude: string (nullable = true)
| | |-- longitude: string (nullable = true)
| | |-- postalCode: string (nullable = true)
| | |-- registeredCountry: string (nullable = true)
| | |-- registeredCountryIsoCode: string (nullable = true)
| | |-- subDivisions: string (nullable = true)
| | |-- subDivisionsIsoCode: string (nullable = true)
| | |-- timeZone: string (nullable = true)
| |-- original: struct (nullable = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- state: string (nullable = true)
| |-- residentialType: string (nullable = true)
| |-- source: string (nullable = true)
| |-- sourceType: string (nullable = true)
| |-- standardized: boolean (nullable = true)
| |-- standardizedDate: string (nullable = true)
| |-- state: string (nullable = true)
| |-- updatedDate: string (nullable = true)
| |-- zipCode: string (nullable = true)
|-- oldCandidateAnonymous: boolean (nullable = true)
|-- optIn: boolean (nullable = true)
|-- optInSMS: boolean (nullable = true)
|-- optInSingleSMS: boolean (nullable = true)
|-- optInTextCommunication: boolean (nullable = true)
|-- optInWhatsApp: boolean (nullable = true)
|-- optedOutByRecruiter: boolean (nullable = true)
|-- partnerTenant: string (nullable = true)
|-- passportNumber: string (nullable = true)
|-- pictureS3Url: string (nullable = true)
|-- pictureUrl: string (nullable = true)
|-- preferredCommunicationMode: struct (nullable = true)
| |-- email: boolean (nullable = true)
| |-- sms: boolean (nullable = true)
|-- preferredContactMethod: string (nullable = true)
|-- preferredFirstName: string (nullable = true)
|-- preferredLocationList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- latLong: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- latLong: string (nullable = true)
| | | |-- location: string (nullable = true)
| | | |-- sourceList: array (nullable = true)
| | | | |-- element: string (containsNull = true)
| | |-- priority: double (nullable = true)
| | |-- sourceList: array (nullable = true)
| | | |-- element: string (containsNull = true)
|-- prefix: string (nullable = true)
|-- previousEmployeeIds: array (nullable = true)
| |-- element: string (containsNull = true)
|-- profileCompletionStage: string (nullable = true)
|-- profileCreatedBy: struct (nullable = true)
| |-- name: string (nullable = true)
| |-- recruiterUserId: string (nullable = true)
|-- profileEnrichmentDate: string (nullable = true)
|-- profileStatus: struct (nullable = true)
| |-- locale: string (nullable = true)
| |-- statusFinal: string (nullable = true)
| |-- statusKey: string (nullable = true)
| |-- statusLanguage: string (nullable = true)
| |-- updatedDate: string (nullable = true)
|-- profileUpdatesByRecruiter: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- name: string (nullable = true)
| | |-- recruiterUserId: string (nullable = true)
|-- profileUrl: string (nullable = true)
|-- pupId: string (nullable = true)
|-- pushedToATSOn: string (nullable = true)
|-- race: string (nullable = true)
|-- recruiterActivityTimestamp: string (nullable = true)
|-- recruiterEmail: string (nullable = true)
|-- recruiterName: string (nullable = true)
|-- recruiterUserId: string (nullable = true)
|-- recruitersUserId: array (nullable = true)
| |-- element: string (containsNull = true)
|-- refNum: string (nullable = true)
|-- referralSource: string (nullable = true)
|-- relocate: boolean (nullable = true)
|-- relocationDate: string (nullable = true)
|-- resumeUrl: string (nullable = true)
|-- scopeAttr: string (nullable = true)
|-- shortTermGoal: string (nullable = true)
|-- singleSmsOptInStatus: string (nullable = true)
|-- skillList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- latestDate: string (nullable = true)
| | |-- mlSkills: struct (nullable = true)
| | | |-- skillSco: string (nullable = true)
| | | |-- skill_imp: string (nullable = true)
| | | |-- skill_map: string (nullable = true)
| | | |-- skill_standard: string (nullable = true)
| | |-- score: double (nullable = true)
| | |-- skillSource: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- type: string (nullable = true)
| | |-- value: string (nullable = true)
| | |-- value_org: string (nullable = true)
|-- skillList_org: string (nullable = true)
|-- skills: array (nullable = true)
| |-- element: string (containsNull = true)
|-- smsOptInStatus: string (nullable = true)
|-- socialProfile: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- createdDate: string (nullable = true)
| | |-- emailId: string (nullable = true)
| | |-- fieldID: string (nullable = true)
| | |-- md5sum: string (nullable = true)
| | |-- priority: double (nullable = true)
| | |-- profile: string (nullable = true)
| | |-- profileLink: string (nullable = true)
| | |-- type: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
|-- standardizedLatestDate: string (nullable = true)
|-- swoopProfileId: string (nullable = true)
|-- teamId: string (nullable = true)
|-- tenantSpecific_additionalFields: struct (nullable = true)
| |-- AVIURMajor: string (nullable = true)
| |-- AnyGEbusinesspreference: string (nullable = true)
| |-- AreyouinterestedinhearingfromGE: string (nullable = true)
| |-- Areyouopentorelocation: string (nullable = true)
| |-- AvailabilityDate: string (nullable = true)
| |-- Available: string (nullable = true)
| |-- AviationAreasofInterest: string (nullable = true)
| |-- BFI_DATA: string (nullable = true)
| |-- BusinessOfInterest: string (nullable = true)
| |-- BusinessSegment: string (nullable = true)
| |-- COSDate: string (nullable = true)
| |-- CandidateType: string (nullable = true)
| |-- CareerLevelInterest: string (nullable = true)
| |-- CollegeorUniversity: string (nullable = true)
| |-- Comments: string (nullable = true)
| |-- Consent_Data: string (nullable = true)
| |-- Consent_ExternalTalentShare: string (nullable = true)
| |-- Consent_GDPR: string (nullable = true)
| |-- ContactMeNow_Jobs_Boomerang: string (nullable = true)
| |-- CountriesDesiredtoWork_GECorp: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- CurrentActivity: string (nullable = true)
| |-- CurrentMatric: string (nullable = true)
| |-- DHEvalBusPref: string (nullable = true)
| |-- DareToLead: string (nullable = true)
| |-- DateLeftGE: string (nullable = true)
| |-- DegreeLevel: string (nullable = true)
| |-- Description: string (nullable = true)
| |-- DesiredCompensationRange: string (nullable = true)
| |-- Expected_Graduation_Date: string (nullable = true)
| |-- FN_group_post: string (nullable = true)
| |-- FN_private_editor: string (nullable = true)
| |-- FieldOfStudy: string (nullable = true)
| |-- FormerGEHRManager: string (nullable = true)
| |-- FormerHRM: string (nullable = true)
| |-- FunctionList: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- FunctionsOfInterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- GEAlumni: string (nullable = true)
| |-- GECareerBand: string (nullable = true)
| |-- GEHiredBusiness: string (nullable = true)
| |-- GEHiredBusinessSegment: string (nullable = true)
| |-- GEHiredFunction: string (nullable = true)
| |-- GEHiredJobFamilySelect: string (nullable = true)
| |-- GEHiredRegion: string (nullable = true)
| |-- GEMgmtLevel: string (nullable = true)
| |-- GEPowerInternLocInterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- GPA: string (nullable = true)
| |-- GPAOVERALLDROPDOWN: string (nullable = true)
| |-- GPA_TEXT: string (nullable = true)
| |-- GradMonth: string (nullable = true)
| |-- GradYear: string (nullable = true)
| |-- GraduationMonth: string (nullable = true)
| |-- GraduationYear: string (nullable = true)
| |-- HaveyouheardofGEbefore: string (nullable = true)
| |-- HiredBusSegment: string (nullable = true)
| |-- HowDidYouHeardAboutThisOpportunity: string (nullable = true)
| |-- HowDidYourHearAboutEvent: string (nullable = true)
| |-- HowWouldYouDescribeYourCareerLevel: string (nullable = true)
| |-- HowWouldYouDescribeYourCurrentCareerLevel: string (nullable = true)
| |-- IConsent: string (nullable = true)
| |-- IWantGEToStayConnectedToMeBy: string (nullable = true)
| |-- IWantGEToTellMeAbout: string (nullable = true)
| |-- IWantToHearFromGE: string (nullable = true)
| |-- IfYouHavePriorGEWorkExperiencePleaseExplainYourRole: string (nullable = true)
| |-- Ifyouareateacherregisteringyourstudents: string (nullable = true)
| |-- InWhichCountriesAreYouCurrentlyAuthorizedToWork: string (nullable = true)
| |-- In_Matric_School: string (nullable = true)
| |-- Interest: string (nullable = true)
| |-- InterestUpdate: string (nullable = true)
| |-- InternalBusinessPreference: string (nullable = true)
| |-- InterviewDate: string (nullable = true)
| |-- Interviewavailabilities: string (nullable = true)
| |-- IsGPARankOnA4Scale: string (nullable = true)
| |-- LMWindCanadaSelectJob: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- LMWindCheckbox: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- LanguagesSpoken: string (nullable = true)
| |-- LastEmailedDate: string (nullable = true)
| |-- LeadershipProgramInterest: string (nullable = true)
| |-- Major: string (nullable = true)
| |-- MajorOrAreaOfStudy: string (nullable = true)
| |-- MalaysiaorSingapore: string (nullable = true)
| |-- MatricCompYear: string (nullable = true)
| |-- Matric_Subjects: string (nullable = true)
| |-- Name_Person_Submitting_on_Behalf_Of_Employee: string (nullable = true)
| |-- NameofSchoolCurrentlyAttended: string (nullable = true)
| |-- NewHireRole: string (nullable = true)
| |-- OpenToRelocation: string (nullable = true)
| |-- OperationallyReadyDate: string (nullable = true)
| |-- OtherCollegeorUniversity: string (nullable = true)
| |-- OtherLanguage: string (nullable = true)
| |-- OverallGPA: string (nullable = true)
| |-- PhDCompleted: string (nullable = true)
| |-- PrefPlace_Business: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- PrefPlace_HighestEducation: string (nullable = true)
| |-- PrefPlace_JobDuties: string (nullable = true)
| |-- PrefPlace_Location: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- QualRegister: string (nullable = true)
| |-- ReadinesstoMove: string (nullable = true)
| |-- Relocation_Employee_Talent_Community: string (nullable = true)
| |-- RelotoWhere: string (nullable = true)
| |-- Richmond_Areas_of_Interest: string (nullable = true)
| |-- Richmond_Certificate_Pursued: string (nullable = true)
| |-- Richmond_Shifts_of_Interest: string (nullable = true)
| |-- SG_Education: string (nullable = true)
| |-- SLG_lng_from: string (nullable = true)
| |-- SLG_lng_to: string (nullable = true)
| |-- SL_lng_from: string (nullable = true)
| |-- SL_lng_to: string (nullable = true)
| |-- SSO_ID: string (nullable = true)
| |-- SalaryExpectations: string (nullable = true)
| |-- Schenectady_Jobs_of_Interest: string (nullable = true)
| |-- SchoolEmail: string (nullable = true)
| |-- SecurityClearance: string (nullable = true)
| |-- Shifts_of_Interest: string (nullable = true)
| |-- SkillBridgeAvailDate: string (nullable = true)
| |-- SkillBridgeCmdAck: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- SkillBridgeGeo: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- SkillBridgeNotes: string (nullable = true)
| |-- TargetedAreasofInterest: string (nullable = true)
| |-- Testfield: string (nullable = true)
| |-- TypeofDegree: string (nullable = true)
| |-- UCAlumni: string (nullable = true)
| |-- UCStatus: string (nullable = true)
| |-- URBusinessPreference: string (nullable = true)
| |-- URGradYear: string (nullable = true)
| |-- URMajor: string (nullable = true)
| |-- URPositionSeeking: string (nullable = true)
| |-- UR_Job_Type_Seeking: string (nullable = true)
| |-- URfunctionofinterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- URroletypeofinterest: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- USExportControl: string (nullable = true)
| |-- USMilAoI: string (nullable = true)
| |-- USMilEASDate: string (nullable = true)
| |-- USMilExperience: string (nullable = true)
| |-- USMilRank: string (nullable = true)
| |-- USMilitaryBranchofService: string (nullable = true)
| |-- USVisaSponsorship: string (nullable = true)
| |-- US_Sponsorship_Needed: string (nullable = true)
| |-- UnderOrPostGradReg: string (nullable = true)
| |-- UniversityOpenText: string (nullable = true)
| |-- University_ModuleSA: string (nullable = true)
| |-- WHATISYOURLEVELOFPARTICIPATION: string (nullable = true)
| |-- Whichfuturebusinessunitsinterestyou: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- YEARINSCHOOL: string (nullable = true)
| |-- YelloRefNumber: string (nullable = true)
| |-- YrsOfExp: string (nullable = true)
| |-- candidatesSMSOptIn: boolean (nullable = true)
| |-- candidatesSingleSMSOptIn: boolean (nullable = true)
| |-- co_speechnotes_plugin_fab_language_selector_id: string (nullable = true)
| |-- currentCollege: string (nullable = true)
| |-- currentOverallGpa: string (nullable = true)
| |-- degree: string (nullable = true)
| |-- degreeLevel: string (nullable = true)
| |-- education: string (nullable = true)
| |-- educationEndDate: string (nullable = true)
| |-- employee_flag: string (nullable = true)
| |-- employee_number: string (nullable = true)
| |-- employee_referral: string (nullable = true)
| |-- enrichmentName: string (nullable = true)
| |-- feedBackInfo: string (nullable = true)
| |-- fieldOfStudy: string (nullable = true)
| |-- field_of_study_italy_Events: string (nullable = true)
| |-- field_of_study_text: string (nullable = true)
| |-- formId: string (nullable = true)
| |-- formType: string (nullable = true)
| |-- gradMonth: string (nullable = true)
| |-- gradYear: string (nullable = true)
| |-- gradyearnew: string (nullable = true)
| |-- hiredBusiness: string (nullable = true)
| |-- hiredBusinessSegment: string (nullable = true)
| |-- initiativesofInterest: string (nullable = true)
| |-- initiativesofInterest_5: string (nullable = true)
| |-- major: string (nullable = true)
| |-- operatingPartners: string (nullable = true)
| |-- optinsinglesms: boolean (nullable = true)
| |-- optinsms: boolean (nullable = true)
| |-- otherCollegeorUniversity: string (nullable = true)
| |-- schoolName: string (nullable = true)
| |-- select_language: string (nullable = true)
| |-- skillSet: string (nullable = true)
| |-- sourcerName: array (nullable = true)
| | |-- element: string (containsNull = true)
| |-- testField: string (nullable = true)
| |-- universityName: string (nullable = true)
| |-- userId: string (nullable = true)
| |-- verifyReg: boolean (nullable = true)
|-- timeZone: string (nullable = true)
|-- topRatedSkills: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- value: string (nullable = true)
|-- trafficSourceList: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- createdDate: string (nullable = true)
| | |-- id: string (nullable = true)
| | |-- rawUrl: string (nullable = true)
| | |-- referralType: string (nullable = true)
| | |-- sourceFullname: string (nullable = true)
| | |-- sourceShortName: string (nullable = true)
| | |-- utmCampaign: string (nullable = true)
| | |-- utmMedium: string (nullable = true)
|-- twitterUrl: string (nullable = true)
|-- updatedDate: string (nullable = true)
|-- userId: string (nullable = true)
|-- userStatus: string (nullable = true)
|-- userType: string (nullable = true)
|-- utids: array (nullable = true)
| |-- element: string (containsNull = true)
|-- veteran: string (nullable = true)
|-- workAuthorization: string (nullable = true)
|-- zipCode: string (nullable = true)
|-- cdc_modifiedon: string (nullable = true)
|-- cdc_operation: string (nullable = true)
|-- updatedOn: string (nullable = true)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1631095205
When i change the scala code to the below (exactly like said in the quick start) the insert itself fails.
```
val sess = Application.spark();
val snapshotDf = sess.read.parquet("s3://bucket/snapshots-test/ge11-drop/")
val cdcSchema1 = SparkUtils.getSchema("s3://bucket/schemas/GE11GLOBAL_candidates-CandidatesList.json")
val cdcDf = sess.read.schema(cdcSchema1).json("s3://bucket/inputs-test/ge11-drop/*")
snapshotDf.createOrReplaceTempView("snapshot")
val snapshotDf2 = snapshotDf.limit(4).withColumn("cdc_pk",lit("0"))
snapshotDf2.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "_id.oid")
.option(HoodieWriteConfig.TABLE_NAME,"GE11")
.mode(SaveMode.Overwrite)
.save("s3://bucket/snapshots-hudi/ge11-drop/snapshot");
```
i start the spark shell as follows
```
spark-shell --driver-memory 1g --executor-memory 4g --executor-cores 1 --driver-cores 1 --conf spark.dynamicAllocation.maxExecutors=2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.kryo.registrator=org.apache.spark.HoodieSparkKryoRegistrar" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --name ravic --packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1 --jars /home/hadoop/jars2/spark-1.0-SNAPSHOT.jar,/home/hadoop/hudi/hudi-release-0.12.3/packaging/hudi-spark-bundle/target/hudi-spark3.3-bundle_2.12-0.12.3.jar
```
Exception (I am unable to get the basic insert working)
```
07-11 15:58:17 ${sys:config.appname} WARN DAGScheduler: Broadcasting large task binary with size 1032.2 KiB
07-11 15:58:18 ${sys:config.appname} ERROR HoodieSparkSqlWriter$: UPSERT failed with errors
org.apache.hudi.exception.HoodieException: Write to Hudi failed
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:148)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
... 59 elided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1635713485
Ok, will recheck. thank you for the update. Will post back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1642011015
I connected with @chandu-1101 and write was failing for one of the record.
As discussed, can you post the sample record for which it was failing. it will help us to fix if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1628857130
The. above is the full stacktrace. Did a fresh run and rewriting it again here. Can these be the reason forthe failure?
1. my src parquet file is 40GB ; I reran with `val snapshotDf3 = snapshotDf2.withColumn("hash", (hash(col("cdc_oid")) %50) )` --> meaning 100 partitions
2. My partition is based out of hash on primary key; max of 100?
3. Executor mem is 3G, driver 1G, executor cores 1, driver 1?
4. Each row is heavy ~ 10kB - 20kB
```
07-10 12:19:14 INBRUS_645d0a45cea0fd64c95d773c_bid_9223370348308073967__merge_ja WARN DAGScheduler: Broadcasting large task binary with size 1053.0 KiB
07-10 12:19:59 INBRUS_645d0a45cea0fd64c95d773c_bid_9223370348308073967__merge_ja ERROR HoodieSparkSqlWriter$: UPSERT failed with errors
org.apache.hudi.exception.HoodieException: Write to Hudi failed
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:148)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
... 59 elided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1630119428
@ ad1happy2go thank you for the reply. I will check these and get back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [BUG] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1655248911
Working as expected thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1628881987
@chandu-1101
- Did you tried with a small parquet dataset? Is it working?
- Are you not seeing any other error in the entire log? It may also due to memory. You can try increasing executor and driver memory if possible.
- If possible, then you can paste the entire log in file and attach to ticket. We can try to look into a clue why its failing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1641941681
Hi,
I found the root cause. The data I had nested Jsons of arrays with null values. And when this is the case the write to hudi fails.
So, given that I have to merge some 500 collections across, now I should write code to recursively traverse every cell of the data frame and remove nulls or substitute with empty/ default values --which is painful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [BUG] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1649646230
@chandu-1101 Created a critical JIRA for this - https://issues.apache.org/jira/browse/HUDI-6589
We will take this up soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [BUG] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1650981047
Wow! Wonderful. Thank you once again. I will put the flag check and get back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 closed issue #9141: [BUG] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 closed issue #9141: [BUG] Example from Hudi Quick start doesnt work!
URL: https://github.com/apache/hudi/issues/9141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1632104122
I think i am able to narrow down the issue to an extent.
1. I tried adding column by column, one by one to the select query and re ran the insert. At one column things broke.
2. Then i adjusted the `limit X` in the sql query (code below) ; and after several iterations i find when `limit 472` things break again! So something weird is happening with the data once the `471st` row is crossed. What is it? I see the Json schema of 471st, 472nd rows similar (https://www.diffchecker.com/QRbXbzd5/)
What is strange to me is::
1. As of now i am using spark-SQL (no hudi) to merge the snapshot (parquet file) with the cdc (json files) and since 1.5y this is going good --meaning existing schema is still intact.
2. Then, why does it fail with Hudi? with an error?
3. One more: if i change the code to add bulk insert, then this insert works (where is the code? previous posts, pl check). But the subsequent merge fails --why? i don't know!
- I am wondering how the companies are using Hudi
- or if CDC merging (meaning CDC dataframe merging with snapshot dataframe in spark) is the wrong usecase to hudi?
- or should the data follow some pattern internally? for things to work? --I have ~5000 db collections to which merge has to be run. I am thinking what can be done if i encounter the exceptions i get, in production?
spark-shell command
```
spark-shell --driver-memory 1g --executor-memory 4g --executor-cores 1 --driver-cores 1 --conf spark.dynamicAllocation.maxExecutors=2 --conf "spark.serializer=org.apache.spark.serializer.KryoSerializer" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf "spark.sql.caseSensitive=true" --conf "spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog" --conf spark.sql.legacy.parquet.int96RebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.int96RebaseModeInWrite=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInRead=CORRECTED --conf spark.sql.legacy.parquet.datetimeRebaseModeInWrite=CORRECTED --name ravic --packages org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1 --jars /home/hadoop/jars2/spark-1.0-SNAPSHOT.jar,/home/hadoop/hudi
/hudi-release-0.12.3/packaging/hudi-spark-bundle/target/hudi-spark3.3-bundle_2.12-0.12.3.jar
```
Code
```
import org.apache.commons.lang3.ClassUtils.getCanonicalName
import org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider
import org.apache.hudi.{DataSourceWriteOptions, QuickstartUtils}
import org.apache.hudi.common.model.{HoodieAvroPayload, HoodieFileFormat, WriteOperationType}
import org.apache.hudi.common.table.HoodieTableConfig
import org.apache.hudi.config.{HoodieBootstrapConfig, HoodieWriteConfig}
import org.apache.hudi.keygen.constant.KeyGeneratorOptions
import java.util
import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
import org.apache.hudi.keygen.{NonpartitionedKeyGenerator, SimpleKeyGenerator}
import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.functions.{col, hash, lit}
import org.apache.hudi.QuickstartUtils._
val sess = spark;
val snapshotDf = sess.read.parquet("s3://p-crm-messaging-v2/snapshots-test/ge11-drop/")
snapshotDf.cache()
snapshotDf.registerTempTable("snapshot")
// 472 is the culprit
val snapshotDf2 = sess.sql("select * from snapshot order by _id.oid limit 472 ")
snapshotDf2.registerTempTable("snapshot2")
val snapshotDf3 = sess.sql("select _id, cdc_pk, addressLogs from snapshot2 ")
snapshotDf3.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "_id.oid")
.option(HoodieWriteConfig.TABLE_NAME, "GE11")
.mode(SaveMode.Overwrite)
.save("s3://p-crm-messaging-v2/snapshots-hudi/ge11-drop/snapshot");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1641452631
Any contributor can help here? Any geek? The seemingly simple insert into HUDI fails!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1639955458
To eliminate the column case sensitiveness issue I renamed all columns to static strings with index-es --Except for ` _id.oid` , `cdc_pk` , and `addressLog` . Also, I printed the schema of `addressLog` and made sure none of the columns repeat. Yet, again the same excption.
NOTE: note that I am running on AWS EMR with only ganglia, Spark selected (no hive/ glue/ hudi are selected)
addressLog column schema
```
|-- addressLogs: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- addressLines: array (nullable = true)
| | | |-- element: string (containsNull = true)
| | |-- city: string (nullable = true)
| | |-- country: string (nullable = true)
| | |-- createdDate: string (nullable = true)
| | |-- fieldId: string (nullable = true)
| | |-- isDerived: boolean (nullable = true)
| | |-- latLong: string (nullable = true)
| | |-- location: string (nullable = true)
| | |-- locationIp: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- continentCode: string (nullable = true)
| | | |-- continentName: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- countryIsoCode: string (nullable = true)
| | | |-- latitude: string (nullable = true)
| | | |-- longitude: string (nullable = true)
| | | |-- postalCode: string (nullable = true)
| | | |-- registeredCountry: string (nullable = true)
| | | |-- registeredCountryIsoCode: string (nullable = true)
| | | |-- subDivisions: string (nullable = true)
| | | |-- subDivisionsIsoCode: string (nullable = true)
| | | |-- timeZone: string (nullable = true)
| | |-- original: struct (nullable = true)
| | | |-- city: string (nullable = true)
| | | |-- country: string (nullable = true)
| | | |-- location: string (nullable = true)
| | | |-- state: string (nullable = true)
| | |-- residentialType: string (nullable = true)
| | |-- source: string (nullable = true)
| | |-- sourceType: string (nullable = true)
| | |-- standardized: boolean (nullable = true)
| | |-- standardizedDate: string (nullable = true)
| | |-- state: string (nullable = true)
| | |-- updatedDate: string (nullable = true)
| | |-- zipCode: string (nullable = true)
```
final code
```
import org.apache.spark.sql.{Column, DataFrame}
import com.phenom.messagingv2.common.Application
import com.phenom.messagingv2.utils.SparkUtils
import org.apache.commons.lang3.ClassUtils.getCanonicalName
import org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider
import org.apache.hudi.{DataSourceWriteOptions, QuickstartUtils}
import org.apache.hudi.common.model.{HoodieAvroPayload, HoodieFileFormat, WriteOperationType}
import org.apache.hudi.common.table.HoodieTableConfig
import org.apache.hudi.config.{HoodieBootstrapConfig, HoodieWriteConfig}
import org.apache.hudi.keygen.constant.KeyGeneratorOptions
import java.util
import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
import org.apache.hudi.keygen.{NonpartitionedKeyGenerator, SimpleKeyGenerator}
import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.functions.{col, hash, lit}
import org.apache.hudi.QuickstartUtils._
def renameColumnsWithIndex(df: DataFrame): DataFrame = {
var df2 = df;
val newColumnNames = (0 to df.columns.length - 1).map(i => s"_index_$i")
println(newColumnNames)
df.columns.zip(newColumnNames).map {
case (oldName, newName) =>
if("cdc_pk".equals(oldName) || oldName.contains("_id") || oldName.contains("oid") || oldName.contains("addressLogs") ) {
// do nothing
}else{
println(oldName+" -> "+ newName);
df2 = df2.withColumnRenamed(oldName, newName)
}
}.foldLeft(df2)((acc, df2) => acc)
}
val sess = spark;
val snapshotDf = sess.read.parquet("s3://bucket/snapshots-test/ge11-drop/")
snapshotDf.cache()
snapshotDf.registerTempTable("snapshot")
// 472 is the culprit
val snapshotDf2 = renameColumnsWithIndex(sess.sql("select * from snapshot order by _id.oid limit 472 ") )
snapshotDf2.registerTempTable("snapshot2")
// val snapshotDf2 = sess.sql("select _id, cdc_pk,eventId, additionalConfig, additionalFields, additionalRequestInfo, address, addressLogs, alertData,alertUrl,applySources,applyTransactionId,atsId, experience from snapshot ")
val snapshotDf3 = sess.sql("select _id, cdc_pk, addressLogs from snapshot2 ")
snapshotDf3.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "_id.oid")
.option(HoodieWriteConfig.TABLE_NAME, "GE11")
.mode(SaveMode.Overwrite)
.save("s3://buket/snapshots-hudi/ge11-drop/snapshot");
```
exception
```
07-18 10:24:07 ${sys:config.appname} ERROR HoodieSparkSqlWriter$: UPSERT failed with errors
org.apache.hudi.exception.HoodieException: Write to Hudi failed
at org.apache.hudi.DefaultSource.createRelation(DefaultSource.scala:153)
at org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:45)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:75)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:84)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:103)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.executeQuery$1(SQLExecution.scala:114)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$7(SQLExecution.scala:139)
at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:107)
at org.apache.spark.sql.execution.SQLExecution$.withTracker(SQLExecution.scala:224)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:139)
at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:245)
at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:138)
at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:779)
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:68)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:100)
at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:96)
at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:177)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:615)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:30)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:591)
at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:96)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:83)
at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:81)
at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:124)
at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:860)
at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:390)
at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:363)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)
... 53 elided
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1635529954
Any update? / pointers? why does this happen ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1628840809
Tried this and it is working good. Can you try it?
```
import org.apache.hudi.QuickstartUtils
import org.apache.hudi.common.model.HoodieAvroPayload
import org.apache.hudi.common.model.WriteOperationType
import org.apache.hudi.config.HoodieWriteConfig
import org.apache.hudi.keygen.constant.KeyGeneratorOptions
import org.apache.spark.SparkConf
import org.apache.spark.api.java.JavaSparkContext
import org.apache.spark.api.java.function.Function
import org.apache.spark.sql.Dataset
import org.apache.spark.sql.Row
import org.apache.spark.sql.SparkSession
import java.util
import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
import org.apache.spark.sql.SaveMode.Append
import org.apache.spark.sql.SaveMode.Overwrite
import org.apache.hudi.DataSourceWriteOptions
val snapshotDf = spark.read.parquet("<Parquet Path>")
snapshotDf.registerTempTable("snapshot");
val snapshotDf2 = spark.sql("select * from snapshot where invoiceid is not null and invoiceid !='' ")
val snapshotDf3 = snapshotDf2.withColumn("hash", lit(col("invoiceid").hashCode() %10) )
snapshotDf3.write.format("hudi").options(QuickstartUtils.getQuickstartWriteConfigs())
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "orderdate")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "invoiceid")
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, "hash")
.option(TBL_NAME.key(), "GE11")
.mode(Overwrite)
.save("file:///tmp/issue_9141");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] chandu-1101 commented on issue #9141: [SUPPORT] Example from Hudi Quick start doesnt work!
Posted by "chandu-1101 (via GitHub)" <gi...@apache.org>.
chandu-1101 commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1643139279
The fix is to change the below
```
"addressLines": [null],
```
to
```
"addressLines": [""],
```
in the source JSON.
code to reproduce the issue.
```
val df1 = spark.read.json(Seq(json1).toDS)
import org.apache.spark.sql.{Column, DataFrame}
import org.apache.commons.lang3.ClassUtils.getCanonicalName
import org.apache.hudi.bootstrap.SparkParquetBootstrapDataProvider
import org.apache.hudi.{DataSourceWriteOptions, QuickstartUtils}
import org.apache.hudi.common.model.{HoodieAvroPayload, HoodieFileFormat, WriteOperationType}
import org.apache.hudi.common.table.HoodieTableConfig
import org.apache.hudi.config.{HoodieBootstrapConfig, HoodieWriteConfig}
import org.apache.hudi.keygen.constant.KeyGeneratorOptions
import java.util
import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
import org.apache.hudi.keygen.{NonpartitionedKeyGenerator, SimpleKeyGenerator}
import org.apache.spark.sql.SaveMode
import org.apache.spark.sql.functions.{col, hash, lit}
import org.apache.hudi.QuickstartUtils._
df1.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option("hoodie.datasource.hive_sync.partition_extractor_class","org.apache.hudi.hive.NonPartitionedExtractor")
.option(DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY, "cdc_pk")
.option(DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY, "_id.oid")
.option(HoodieWriteConfig.TABLE_NAME, "GE11")
.mode(SaveMode.Overwrite)
.save("s3://bucket/snapshots-hudi/ge11-drop/snapshot1");
```
The record that doesnt work
```
{
"_id": {
"oid": "1"
},
"cdc_pk": "45",
"addressLogs": [{
"createdDate": "2021-09-06T17:17:41.576Z",
"fieldId": "eb4b6bd9-1fc0-4d38-b2d4-4cba87bb65a4",
"isDerived": false,
"location": "hyderabad (HC) PL",
"original": {
"location": "hyderabad (HC) PL"
},
"source": "p2",
"standardized": false,
"updatedDate": "2023-06-29T20:44:26.788Z"
}, {
"addressLines": [null],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-29T20:44:26.788Z",
"fieldId": "1beefa35-7d08-4ca7-9fe1-88e59abb4c89",
"isDerived": false,
"location": "hyderabad, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, Srilanka"
},
"residentialType": "HOME",
"source": "p2",
"standardized": false,
"updatedDate": "2023-06-29T20:44:26.788Z"
}, {
"addressLines": ["raddison,Business Park,cly B-"],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-29T20:44:26.788Z",
"fieldId": "42720793-1920-4a35-9e3e-23f91e00341e",
"isDerived": false,
"location": "hyderabad, TN, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, TN, Srilanka",
"state": "TN"
},
"residentialType": "WORK",
"source": "p2",
"standardized": false,
"state": "TN",
"updatedDate": "2023-06-29T20:44:26.788Z",
"zipCode": "02-583"
}, {
"addressLines": [null],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-28T19:48:31.948Z",
"fieldId": "7b56cdae-fbbc-4dd4-996e-6214b590db4a",
"isDerived": false,
"location": "hyderabad, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, Srilanka"
},
"residentialType": "HOME",
"source": "p2",
"standardized": false,
"updatedDate": "2023-06-28T19:48:31.948Z"
}, {
"addressLines": ["raddison,Business Park,cly B-"],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-28T19:48:31.948Z",
"fieldId": "27e67381-c688-4879-a0a4-319ae051dca8",
"isDerived": false,
"location": "hyderabad, TN, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, TN, Srilanka",
"state": "TN"
},
"residentialType": "WORK",
"source": "p2",
"standardized": false,
"state": "TN",
"updatedDate": "2023-06-28T19:48:31.948Z",
"zipCode": "02-583"
}, {
"addressLines": [null],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2021-11-26T14:49:38.305Z",
"fieldId": "0352928b-1a42-40d7-81fd-cf711a2ecda1",
"isDerived": false,
"location": "hyderabad, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, Srilanka"
},
"residentialType": "HOME",
"source": "p4",
"standardized": false,
"updatedDate": "2022-03-24T13:52:11.876Z"
}, {
"addressLines": ["raddison,Business Park,cly B-"],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2021-11-26T14:49:38.305Z",
"fieldId": "058fc73f-559c-414c-8dfb-9ad830fbbdf3",
"isDerived": false,
"location": "hyderabad, TN, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, TN, Srilanka",
"state": "TN"
},
"residentialType": "WORK",
"source": "p4",
"standardized": false,
"state": "TN",
"updatedDate": "2022-03-24T13:52:11.876Z",
"zipCode": "02-583"
}, {
"createdDate": "2021-09-03T16:36:57.802Z",
"fieldId": "c8c7e9c7-f7ad-4df4-90d2-5d07eeaa141f",
"isDerived": false,
"location": "hyderabad (HC) PL",
"original": {
"location": "hyderabad (HC) PL"
},
"source": "p4",
"standardized": false,
"updatedDate": "2022-03-24T13:52:11.876Z"
}, {
"country": "Srilanka",
"createdDate": "2020-01-29T16:14:00.050Z",
"fieldId": "4fbcc142-565d-4aa4-af00-6daa807dd951",
"isDerived": false,
"location": "Srilanka",
"locationIp": {
"city": "Amsterdam",
"continentCode": "EU",
"continentName": "Europe",
"country": "Netherlands",
"countryIsoCode": "NL",
"latitude": "52.3759",
"longitude": "4.8975",
"postalCode": "1012",
"registeredCountry": "United Kingdom",
"registeredCountryIsoCode": "GB",
"subDivisions": "North Holland",
"subDivisionsIsoCode": "NH",
"timeZone": "Europe/Amsterdam"
},
"original": {
"country": "Srilanka",
"location": "Srilanka"
},
"source": "p9",
"standardized": false,
"updatedDate": "2023-02-25T19:31:23.901Z"
}, {
"addressLines": [null],
"city": "hyderabad",
"country": "POL",
"createdDate": "2021-08-10T16:34:32.662Z",
"fieldId": "48318942-e268-4d66-8084-e19e302a73d7",
"isDerived": false,
"location": "hyderabad, POL",
"original": {
"city": "hyderabad",
"country": "POL",
"location": "hyderabad, POL"
},
"source": "p11",
"standardized": false,
"updatedDate": "2021-08-11T10:47:02.326Z"
}]
}
```
The corrected record that works
```
{
"_id": {
"oid": "1"
},
"cdc_pk": "45",
"addressLogs": [{
"createdDate": "2021-09-06T17:17:41.576Z",
"fieldId": "eb4b6bd9-1fc0-4d38-b2d4-4cba87bb65a4",
"isDerived": false,
"location": "hyderabad (HC) PL",
"original": {
"location": "hyderabad (HC) PL"
},
"source": "p2",
"standardized": false,
"updatedDate": "2023-06-29T20:44:26.788Z"
}, {
"addressLines": [""],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-29T20:44:26.788Z",
"fieldId": "1beefa35-7d08-4ca7-9fe1-88e59abb4c89",
"isDerived": false,
"location": "hyderabad, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, Srilanka"
},
"residentialType": "HOME",
"source": "p2",
"standardized": false,
"updatedDate": "2023-06-29T20:44:26.788Z"
}, {
"addressLines": ["raddison,Business Park,cly B-"],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-29T20:44:26.788Z",
"fieldId": "42720793-1920-4a35-9e3e-23f91e00341e",
"isDerived": false,
"location": "hyderabad, TN, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, TN, Srilanka",
"state": "TN"
},
"residentialType": "WORK",
"source": "p2",
"standardized": false,
"state": "TN",
"updatedDate": "2023-06-29T20:44:26.788Z",
"zipCode": "02-583"
}, {
"addressLines": [""],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-28T19:48:31.948Z",
"fieldId": "7b56cdae-fbbc-4dd4-996e-6214b590db4a",
"isDerived": false,
"location": "hyderabad, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, Srilanka"
},
"residentialType": "HOME",
"source": "p2",
"standardized": false,
"updatedDate": "2023-06-28T19:48:31.948Z"
}, {
"addressLines": ["raddison,Business Park,cly B-"],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2023-06-28T19:48:31.948Z",
"fieldId": "27e67381-c688-4879-a0a4-319ae051dca8",
"isDerived": false,
"location": "hyderabad, TN, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, TN, Srilanka",
"state": "TN"
},
"residentialType": "WORK",
"source": "p2",
"standardized": false,
"state": "TN",
"updatedDate": "2023-06-28T19:48:31.948Z",
"zipCode": "02-583"
}, {
"addressLines": [""],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2021-11-26T14:49:38.305Z",
"fieldId": "0352928b-1a42-40d7-81fd-cf711a2ecda1",
"isDerived": false,
"location": "hyderabad, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, Srilanka"
},
"residentialType": "HOME",
"source": "p4",
"standardized": false,
"updatedDate": "2022-03-24T13:52:11.876Z"
}, {
"addressLines": ["raddison,Business Park,cly B-"],
"city": "hyderabad",
"country": "Srilanka",
"createdDate": "2021-11-26T14:49:38.305Z",
"fieldId": "058fc73f-559c-414c-8dfb-9ad830fbbdf3",
"isDerived": false,
"location": "hyderabad, TN, Srilanka",
"original": {
"city": "hyderabad",
"country": "Srilanka",
"location": "hyderabad, TN, Srilanka",
"state": "TN"
},
"residentialType": "WORK",
"source": "p4",
"standardized": false,
"state": "TN",
"updatedDate": "2022-03-24T13:52:11.876Z",
"zipCode": "02-583"
}, {
"createdDate": "2021-09-03T16:36:57.802Z",
"fieldId": "c8c7e9c7-f7ad-4df4-90d2-5d07eeaa141f",
"isDerived": false,
"location": "hyderabad (HC) PL",
"original": {
"location": "hyderabad (HC) PL"
},
"source": "p4",
"standardized": false,
"updatedDate": "2022-03-24T13:52:11.876Z"
}, {
"country": "Srilanka",
"createdDate": "2020-01-29T16:14:00.050Z",
"fieldId": "4fbcc142-565d-4aa4-af00-6daa807dd951",
"isDerived": false,
"location": "Srilanka",
"locationIp": {
"city": "Amsterdam",
"continentCode": "EU",
"continentName": "Europe",
"country": "Netherlands",
"countryIsoCode": "NL",
"latitude": "52.3759",
"longitude": "4.8975",
"postalCode": "1012",
"registeredCountry": "United Kingdom",
"registeredCountryIsoCode": "GB",
"subDivisions": "North Holland",
"subDivisionsIsoCode": "NH",
"timeZone": "Europe/Amsterdam"
},
"original": {
"country": "Srilanka",
"location": "Srilanka"
},
"source": "p9",
"standardized": false,
"updatedDate": "2023-02-25T19:31:23.901Z"
}, {
"addressLines": [""],
"city": "hyderabad",
"country": "POL",
"createdDate": "2021-08-10T16:34:32.662Z",
"fieldId": "48318942-e268-4d66-8084-e19e302a73d7",
"isDerived": false,
"location": "hyderabad, POL",
"original": {
"city": "hyderabad",
"country": "POL",
"location": "hyderabad, POL"
},
"source": "p11",
"standardized": false,
"updatedDate": "2021-08-11T10:47:02.326Z"
}]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #9141: [BUG] Example from Hudi Quick start doesnt work!
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #9141:
URL: https://github.com/apache/hudi/issues/9141#issuecomment-1650216822
@chandu-1101 Upon further investigation and debugging, it has been determined that to address the issue related to Avro-parquet compatibility and allow arrays with null elements, you need to set the Spark configuration parameter spark.hadoop.parquet.avro.write-old-list-structure to false.
This configuration parameter controls the behavior of how Avro arrays with null elements are written to Parquet format. By default, Avro arrays with null elements are written in a way that preserves their internal structure, which can cause compatibility problems with certain tools. By setting spark.hadoop.parquet.avro.write-old-list-structure to false, you enable support for arrays with null elements and ensure they are handled correctly during the write process.
This was not a Hudi issue. I was able to insert the record you pasted by just setting this --conf 'spark.hadoop.parquet.avro.write-old-list-structure=false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org