You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hudi.apache.org by lamberken <la...@163.com> on 2019/12/25 13:32:36 UTC

Re:CHD6.3 hadoop3 quckstart


Hi,


It's because there are spaces in the basePath, can you try to use this?
val basePath = "file:///tmp/hudi_cow_table”


best, 
lamber-ken


On 12/25/2019 20:50，965147871@qq.com<96...@qq.com> wrote：

hi,all


The environment I use is CDH6.3,
Use hadoop3 maven dependency to compile hudi,  <hadoop.version>3.0.0</hadoop.version>
execute quickstart,
bin / spark-shell --jars /home/t3cx/apps/hudi/hudi-spark-bundle-0.5.1-SNAPSHOT.jar --conf 'spark.serializer = org.apache.spark.serializer.KryoSerializer'

import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._

val tableName = "hudi_cow_table"
val basePath = "file: /// tmp / hudi_cow_table"
val dataGen = new DataGenerator

val inserts = convertToStringList (dataGen.generateInserts (10))
val df = spark.read.json (spark.sparkContext.parallelize (inserts, 2))
df.write.format ("org.apache.hudi").
options (getQuickstartWriteConfigs).
option (PRECOMBINE_FIELD_OPT_KEY, "ts").
option (RECORDKEY_FIELD_OPT_KEY, "uuid").
option (PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
option (TABLE_NAME, tableName).
mode (Overwrite).
save (basePath);

/ tmp / hudi_cow_table / No data in this directory

please help


965147871@qq.com

Re: Re:CHD6.3 hadoop3 quckstart

Posted by "965147871@qq.com" <96...@qq.com>.

hi,

Sorry, no space was noticed when the demo was copied to the email. I did not have spaces when I actually executed it.
Therefore, it seems that it is not caused by spaces. this / tmp / hudi_cow_table this file directory is generated,  but there is no file data in it.

thanks
liujh


965147871@qq.com
 
From: lamberken
Date: 2019-12-25 21:32
To: dev@hudi.apache.org
Subject: Re:CHD6.3 hadoop3 quckstart
 
 
Hi,
 
 
It's because there are spaces in the basePath, can you try to use this?
val basePath = "file:///tmp/hudi_cow_table”
 
 
best, 
lamber-ken
 
 
On 12/25/2019 20:50，965147871@qq.com<96...@qq.com> wrote：
 
hi,all
 
 
The environment I use is CDH6.3,
Use hadoop3 maven dependency to compile hudi,  <hadoop.version>3.0.0</hadoop.version>
execute quickstart,
bin / spark-shell --jars /home/t3cx/apps/hudi/hudi-spark-bundle-0.5.1-SNAPSHOT.jar --conf 'spark.serializer = org.apache.spark.serializer.KryoSerializer'
 
import org.apache.hudi.QuickstartUtils._
import scala.collection.JavaConversions._
import org.apache.spark.sql.SaveMode._
import org.apache.hudi.DataSourceReadOptions._
import org.apache.hudi.DataSourceWriteOptions._
import org.apache.hudi.config.HoodieWriteConfig._
 
val tableName = "hudi_cow_table"
val basePath = "file: /// tmp / hudi_cow_table"
val dataGen = new DataGenerator
 
val inserts = convertToStringList (dataGen.generateInserts (10))
val df = spark.read.json (spark.sparkContext.parallelize (inserts, 2))
df.write.format ("org.apache.hudi").
options (getQuickstartWriteConfigs).
option (PRECOMBINE_FIELD_OPT_KEY, "ts").
option (RECORDKEY_FIELD_OPT_KEY, "uuid").
option (PARTITIONPATH_FIELD_OPT_KEY, "partitionpath").
option (TABLE_NAME, tableName).
mode (Overwrite).
save (basePath);
 
/ tmp / hudi_cow_table / No data in this directory
 
please help
 
 
965147871@qq.com