You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@amaterasu.apache.org by Arun Manivannan <ar...@arunma.com> on 2019/01/12 14:20:29 UTC

Unable to read config with konf

All,

As part of Amaterasu 52, I am trying to come up with a format for datasets
configuration. Here's what I have

datasets:
  hive:
    - key: transactions
      uri: /user/somepath
      format: parquet
      database: transations_daily
      table: transx

    - key: second_transactions
      uri: /seconduser/somepath
      format: avro
      database: transations_monthly
      table: avro_table
  file:
    - key: users
      uri: s3://filestore
      format: parquet
      mode: overwrite

For the konf spec, I have


object DataSetsSpec: ConfigSpec("datasets"){
    val fileDataSets by optional<List<FileDS>>(default = emptyList())
    val hive by optional<List<HiveDS>>(default = emptyList())
}

object FileDS: ConfigSpec ("file"){
    val key by required<String>(description = "Name of the dataset")
    val uri by required<String>(description = "Target path of the file
with the protocol qualifier")
    val format by optional<String>("parquet", "File format", "Format
in which the file must be writte")
    val mode by optional<String>("append", "Save Mode", "Mode in which
the file would be written - (overwrite, append)")
}

object HiveDS: ConfigSpec ("hive"){
    val key by required<String>()
    val uri by required<String>()
    val database by required<String>()
    val table by required<String>()
    val format by optional<String>("parquet", "File format", "Format
in which the file must be writte")
    val mode by optional<String>("append", "Save Mode", "Mode in which
the file would be written - (overwrite, append)")
}



New to Kotlin and Konf, I have been breaking my head over this for a couple
of hours now and finally created the issue
https://github.com/uchuhimo/konf/issues/17

Please let me know if you have any hints around this.

Cheers,
Arun