You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@griffin.apache.org by GitBox <gi...@apache.org> on 2020/04/06 15:30:47 UTC
[GitHub] [griffin] chitralverma opened a new pull request #569:
[GRIFFIN-326] New Data Connector for Elasticsearch
chitralverma opened a new pull request #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569
**What changes were proposed in this pull request?**
This ticket proposes the following changes,
- Deprecate the current implementation in favour of the direct implementation in the official [elasticsearch-hadoop](https://github.com/elastic/elasticsearch-hadoop/tree/master/spark/sql-20) library.
- This library is built on DataSource API built on spark 2.2.x+ and thus brings support for filter pushdowns, column pruning, unified read and write and additional optimizations.
- Many configuration options are available for ES connectivity, [check here](https://github.com/elastic/elasticsearch-hadoop/blob/master/mr/src/main/java/org/elasticsearch/hadoop/cfg/ConfigurationOptions.java).
- Any filters can be applied as expressions directly on the data frame and are pushed automatically to the source.
**Does this PR introduce any user-facing change?**
Yes. As mentioned above, the old connector has been deprecated and config structure for Elasticsearch data connector has changed now.
**How was this patch tested?**
Griffin test suite and additional unit test cases
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] chitralverma commented on issue #569: [GRIFFIN-326] New
Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
chitralverma commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612780594
This is a source connector, not a sink. I'm working on it separately
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] DragonPrince1992 commented on issue #569: [GRIFFIN-326]
New Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
DragonPrince1992 commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612781759
I could not see healthy graphics ,how to solve it
------------------ 原始邮件 ------------------
发件人: "Chitral Verma"<notifications@github.com>;
发送时间: 2020年4月13日(星期一) 下午3:03
收件人: "apache/griffin"<griffin@noreply.github.com>;
抄送: "苍龙七宿"<1982037728@qq.com>;"Comment"<comment@noreply.github.com>;
主题: Re: [apache/griffin] [GRIFFIN-326] New Data Connector for Elasticsearch (#569)
This is a source connector, not a sink. I'm working on it separately
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] guoyuepeng edited a comment on issue #569: [GRIFFIN-326]
New Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
guoyuepeng edited a comment on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612545643
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received
@wankunde
The same issue as before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] wankunde commented on issue #569: [GRIFFIN-326] New Data
Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
wankunde commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612574030
@chitralverma @guoyuepeng
`FileBasedDataConnectorTest` will generate some test data files in `file://${getClass.getResource("/").getPath}` directory, I am not sure travis has some restrictions on this directory. If this problem occurs again, @chitralverma could put those files directly into test resources directory.
@chitralverma could you update your PR with little change and try again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] chitralverma commented on issue #569: [GRIFFIN-326] New
Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
chitralverma commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-610206820
@wankunde @guoyuepeng Can you please review this. Thanks.
Also, I think the build is stuck?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] chitralverma commented on issue #569: [GRIFFIN-326] New
Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
chitralverma commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612782390
also, you have to use `es.nodes` not `es.hostname`
This PR has not been closed yet
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] DragonPrince1992 commented on issue #569: [GRIFFIN-326]
New Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
DragonPrince1992 commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612784719
I can't get it. Can I fix this as configuration? Under file \griffin-master\measure\src\main\resources\env-batch.json
------------------ 原始邮件 ------------------
发件人: "Chitral Verma"<notifications@github.com>;
发送时间: 2020年4月13日(星期一) 下午3:09
收件人: "apache/griffin"<griffin@noreply.github.com>;
抄送: "苍龙七宿"<1982037728@qq.com>;"Comment"<comment@noreply.github.com>;
主题: Re: [apache/griffin] [GRIFFIN-326] New Data Connector for Elasticsearch (#569)
also, you have to use es.nodes not es.hostname
This PR has not been closed yet
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] guoyuepeng commented on issue #569: [GRIFFIN-326] New
Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
guoyuepeng commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612545643
Not sure what happened on CI, let me check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] DragonPrince1992 commented on issue #569: [GRIFFIN-326]
New Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
DragonPrince1992 commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612790960
Can I add you a WeChat?It's easier to communicate.
------------------ 原始邮件 ------------------
发件人: "Chitral Verma"<notifications@github.com>;
发送时间: 2020年4月13日(星期一) 下午3:09
收件人: "apache/griffin"<griffin@noreply.github.com>;
抄送: "苍龙七宿"<1982037728@qq.com>;"Comment"<comment@noreply.github.com>;
主题: Re: [apache/griffin] [GRIFFIN-326] New Data Connector for Elasticsearch (#569)
also, you have to use es.nodes not es.hostname
This PR has not been closed yet
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [griffin] DragonPrince1992 commented on issue #569: [GRIFFIN-326]
New Data Connector for Elasticsearch
Posted by GitBox <gi...@apache.org>.
DragonPrince1992 commented on issue #569: [GRIFFIN-326] New Data Connector for Elasticsearch
URL: https://github.com/apache/griffin/pull/569#issuecomment-612779282
es的hostname都配置正确，在job执行完成之后，es里边查询数据是没有的，这个是怎么回事？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services