You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@gobblin.apache.org by Jay Sen <ja...@apache.org> on 2019/05/11 01:00:04 UTC

Hive source is file based extractor ?

Hi Gobblin Team,

I notice that to run hive to hive copy, even when job config mentions
source  and target hive metastore details, it does not work until you
explicitly specify source.filebased.fs.uri ( for source) and writer.fs.uri
(for target ) to point to hdfs://

the above are File Extractor based propeties, wonder why its used while
trying to figure out source and target FileSystem in CopySource.
      final FileSystem sourceFs = HadoopUtils.getSourceFileSystem(state);
      final FileSystem targetFs = HadoopUtils.getWriterFileSystem(state, 1,
0);

wont the source and target hive metastore enough to say its hdfs source and
target ?

Thanks
Jay