You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/05/19 22:31:19 UTC
[GitHub] [iceberg] jeff1evesque opened a new issue, #4820: Implement 'print' connector with sliding window sink
jeff1evesque opened a new issue, #4820:
URL: https://github.com/apache/iceberg/issues/4820
I'm trying to combine the [`SlidingWindows/sliding-windows.py`](https://github.com/aws-samples/pyflink-getting-started/blob/main/pyflink-examples/SlidingWindows/sliding-windows.py#L88-L97) example, with the [`GettingStarted/getting-started.py`](https://github.com/aws-samples/pyflink-getting-started/blob/main/pyflink-examples/GettingStarted/getting-started.py). Specifically, I've created my own variant of [`sliding_window.py`](sliding_window.py), with an associated [`datagen/stock.py`](stock.py). However, when I run my variation, nothing happens in the Pycharm output:
However, if I comment out the first [`sliding_window_table`](https://github.com/jeff1evesque/kinesis-analytics-demo/blob/master/flink/sliding_window.py#L155-L167) assignment, and uncomment the second [`sliding_window_table`](https://github.com/jeff1evesque/kinesis-analytics-demo/blob/master/flink/sliding_window.py#L168) assignment:
```python
#sliding_window_table = (
# input_table.window(
# Slide.over(sliding_window_over)
# .every(sliding_window_every)
# .on(sliding_window_on)
# .alias(sliding_window_alias)
# )
# .group_by('ticker, {}'.format(sliding_window_alias))
# .select('ticker, price.min as p, {}.end as t'.format(
# sliding_window_alias,
# sliding_window_on
# ))
#)
sliding_window_table = input_table.select('ticker, price, utc')
```
Then, the console is able to capture output:
However, my desire is to combine both the AWS provided sliding window example with the `print` connector example. Once I can get that baseline setup, I plan on deploying pyflink to AWS Kinesis Data Analytics. From there, I intend to join pyflink streaming data with potentially s3 data using Apache Iceberg. However, I may try to utilize Apache iceberg locally using the above PyCharm pyflink setup, before promoting the deployment package to Kinesis Analytics. Could someone with enough free cycles help me with my base case, and potentially set me off on the right foot with joining s3 data using Apache Iceberg?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4820: Implement 'print' connector with sliding window sink
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4820:
URL: https://github.com/apache/iceberg/issues/4820#issuecomment-1133468295
I don't know if I'll have much time to assist coming up, but the `IllegalReflectiveAccess` warning from the first screenshot is not something to be concerned about.
That is actually a very normal log, particularly in the big data world, and should be ignored.
As for why the first one appears not to be running, can you look at the Flink UI and check? Maybe the joins aren't matching up? I'm not at all familiar with the AWS example code, and I don't think any of this necessarily has Iceberg in it at the moment (if I'm not mistaken), so unfortunately I probably wouldn't be of much use. =/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jeff1evesque closed issue #4820: Implement 'print' connector with sliding window sink
Posted by GitBox <gi...@apache.org>.
jeff1evesque closed issue #4820: Implement 'print' connector with sliding window sink
URL: https://github.com/apache/iceberg/issues/4820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jeff1evesque commented on issue #4820: Implement 'print' connector with sliding window sink
Posted by GitBox <gi...@apache.org>.
jeff1evesque commented on issue #4820:
URL: https://github.com/apache/iceberg/issues/4820#issuecomment-1133481892
I'm super new to flink. I just googled Flink UI (web interface), and it seems like a nice [tool to monitor jobs](https://docs.aws.amazon.com/emr/latest/ReleaseGuide/flink-web-interface.html). I did some preliminary search on the tool, and found ways to configure it with [standard Flink UI](https://stackoverflow.com/questions/46988499/flink-webui-when-running-from-ide). I'll check to see if PyFlink includes it, or how to configure it within my PyCharm setup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] jeff1evesque commented on issue #4820: Implement 'print' connector with sliding window sink
Posted by GitBox <gi...@apache.org>.
jeff1evesque commented on issue #4820:
URL: https://github.com/apache/iceberg/issues/4820#issuecomment-1136424166
@kbendick -- yessir, and thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] kbendick commented on issue #4820: Implement 'print' connector with sliding window sink
Posted by GitBox <gi...@apache.org>.
kbendick commented on issue #4820:
URL: https://github.com/apache/iceberg/issues/4820#issuecomment-1136403891
Hi @jeff1evesque. I noticed in your fork that you've had some activity since this. Is this something we can close? I think this is more Flink specific than Iceberg specific.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org