You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@amaterasu.apache.org by Arun Manivannan <ar...@arunma.com> on 2019/01/12 14:20:29 UTC
Unable to read config with konf
All,
As part of Amaterasu 52, I am trying to come up with a format for datasets
configuration. Here's what I have
datasets:
hive:
- key: transactions
uri: /user/somepath
format: parquet
database: transations_daily
table: transx
- key: second_transactions
uri: /seconduser/somepath
format: avro
database: transations_monthly
table: avro_table
file:
- key: users
uri: s3://filestore
format: parquet
mode: overwrite
For the konf spec, I have
object DataSetsSpec: ConfigSpec("datasets"){
val fileDataSets by optional<List<FileDS>>(default = emptyList())
val hive by optional<List<HiveDS>>(default = emptyList())
}
object FileDS: ConfigSpec ("file"){
val key by required<String>(description = "Name of the dataset")
val uri by required<String>(description = "Target path of the file
with the protocol qualifier")
val format by optional<String>("parquet", "File format", "Format
in which the file must be writte")
val mode by optional<String>("append", "Save Mode", "Mode in which
the file would be written - (overwrite, append)")
}
object HiveDS: ConfigSpec ("hive"){
val key by required<String>()
val uri by required<String>()
val database by required<String>()
val table by required<String>()
val format by optional<String>("parquet", "File format", "Format
in which the file must be writte")
val mode by optional<String>("append", "Save Mode", "Mode in which
the file would be written - (overwrite, append)")
}
New to Kotlin and Konf, I have been breaking my head over this for a couple
of hours now and finally created the issue
https://github.com/uchuhimo/konf/issues/17
Please let me know if you have any hints around this.
Cheers,
Arun