You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "alberttwong (via GitHub)" <gi...@apache.org> on 2024/02/19 05:57:41 UTC
[I] [SUPPORT] Hudi wants to write the database in s3://datalake [hudi]
alberttwong opened a new issue, #10695:
URL: https://github.com/apache/hudi/issues/10695
Using the NYC taxi data. I don't define the bucket s3://datalake anywhere in my config files.
```
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
import scala.collection.JavaConversions._
val df = spark.read.parquet("s3://huditest/green_tripdata_2023-01.parquet")
val databaseName = "hudi_sample"
val tableName = "hudi_coders_hive"
val basePath = "s3a://huditest/hudi_coders"
df.write.format("hudi").
option(org.apache.hudi.config.HoodieWriteConfig.TABLE_NAME, tableName).
option(RECORDKEY_FIELD_OPT_KEY, "lpep_pickup_datetime").
option(PRECOMBINE_FIELD_OPT_KEY, "lpep_pickup_datetime").
option("hoodie.datasource.hive_sync.enable", "true").
option("hoodie.datasource.hive_sync.mode", "hms").
option("hoodie.datasource.hive_sync.database", databaseName).
option("hoodie.datasource.hive_sync.table", tableName).
option("hoodie.datasource.hive_sync.metastore.uris", "thrift://hive-metastore:9083").
mode(Overwrite).
save(basePath)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi wants to write the database in s3://datalake [hudi]
Posted by "alberttwong (via GitHub)" <gi...@apache.org>.
alberttwong closed issue #10695: [SUPPORT] Hudi wants to write the database in s3://datalake
URL: https://github.com/apache/hudi/issues/10695
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi wants to write the database in s3://datalake [hudi]
Posted by "alberttwong (via GitHub)" <gi...@apache.org>.
alberttwong commented on issue #10695:
URL: https://github.com/apache/hudi/issues/10695#issuecomment-1951752964
Answer. have you add another option
```
import org.apache.spark.sql.functions._
import org.apache.spark.sql.types._
import org.apache.spark.sql.Row
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
import scala.collection.JavaConversions._
val df = spark.read.parquet("s3://huditest/green_tripdata_2023-01.parquet")
val databaseName = "hudi_sample"
val tableName = "hudi_coders_hive"
val basePath = "s3a://huditest/hudi_coders"
df.write.format("hudi").
option(org.apache.hudi.config.HoodieWriteConfig.TABLE_NAME, tableName).
option(RECORDKEY_FIELD_OPT_KEY, "lpep_pickup_datetime").
option(PRECOMBINE_FIELD_OPT_KEY, "lpep_pickup_datetime").
option("hoodie.datasource.hive_sync.enable", "true").
option("hoodie.datasource.hive_sync.mode", "hms").
option("hoodie.datasource.hive_sync.database", databaseName).
option("hoodie.datasource.hive_sync.table", tableName).
option("hoodie.datasource.hive_sync.metastore.uris", "thrift://hive-metastore:9083").
option("fs.defaultFS", "s3://huditest/").
mode(Overwrite).
save(basePath)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org