You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Matt Casters <ma...@neotechnology.com> on 2021/10/12 08:02:27 UTC
IO Connector
Dear Beam devs,
I took it upon myself to write a Neo4jIO connector to allow us to read from
and write to Neo4j graphs.
All in all it should be a fairly easy task but I'm actually struggling
despite my extensive experience with both Beam and Neo4j.
The blocking issues vary a bit but seem to revolve around my inability to
get my Beam development environment in working order.
1) Setting up my Beam development
<https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch>
environment
for IDEA 2021.2 is something that's going wrong, probably around Gradle
configurations.
2) I can't get Google Auto to work in my IDE (IDEA) because of what seems
outdated documentation
<https://github.com/google/auto/blob/master/value/userguide/index.md>?
3) Since I'm obviously planning to generate a PR at the end of this
exercise: what is the suggested code format for Java in the Beam project?
Thanks for any answers or suggestions. I'd be happy to help update the
docs once I figured out where things possibly go wrong.
I'm thinking of a README.md in sdks/java/io to point folks in the right
direction.
Cheers,
Matt
Re: IO Connector
Posted by Matt Casters <ma...@neo4j.com>.
Hi Alexey,
I'm running on Ubuntu 20.04.3 LTS with IntelliJ IDEA 2021.2. None of the
aforementioned documentation seems to apply.
This forces me into a wild goose chase in figuring out which part of my
environment is not matching what you wonderful people were expecting.
Is gradle version 4.4.1 correct? Is there anything specific I need to do
for the newer IDEA version? I have no idea and so I'm fairly stuck. Once
I figure it out I'll document it somewhere.
Regarding the Neo4j IO connectors: several people
<https://issues.apache.org/jira/browse/BEAM-1857> have at least had the
desire to get started with it but I guess they all gave up. Looking at that
Spotless command, who could really blame them :-)
In all seriousness, this is probably a series of small roadblocks but
without additional information I'm just stuck.
Cheers,
Matt
On Tue, Oct 12, 2021 at 3:08 PM Alexey Romanenko <ar...@gmail.com>
wrote:
> Hi Matt,
>
> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com>
> wrote:
>
> 1) Setting up my Beam development
> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
> for IDEA 2021.2 is something that's going wrong, probably around Gradle
> configurations.
>
> 2) I can't get Google Auto to work in my IDE (IDEA) because of what seems
> outdated documentation
> <https://github.com/google/auto/blob/master/value/userguide/index.md>?
>
>
> Could you elaborate more what is wrong with 1) and 2) ?
>
> 3) Since I'm obviously planning to generate a PR at the end of this
> exercise: what is the suggested code format for Java in the Beam project?
>
>
> Please, run this command before committing your changes:
>
> ./gradlew spotlessApply && ./gradlew
> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
> spotbugsMain compileJava compileTestJava
>
> To save a time, run it only against a package where you did the changes.
>
> —
> Alexey
>
>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: IO Connector
Posted by Brian Hulette <bh...@google.com>.
> Is gradle version 4.4.1 correct?
If you use the gradle wrapper (./gradlew) it will download and use gradle
6.9.1 [1]. I don't know if gradle 4.4.1 will work.
Brian
[1]
https://github.com/apache/beam/blob/4b11efdf96ea4a471e078ec49906c40ef033aafb/gradle/wrapper/gradle-wrapper.properties#L3
On Tue, Oct 12, 2021 at 12:59 PM Evan Galpin <ev...@gmail.com> wrote:
> @Matt have you tried any of the "IDE Tasks" available through gradle?
> "./gradlew tasks" from beam top-level will list available tasks, and the
> IDE Tasks subsection includes tasks specific to trying to bootstrap or
> clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea"
> should set up the project files for use in Intellij. There's also
> "./gradlew cleanIdea" which may be helpful to you.
>
> With respect to Google Auto, I've experienced plenty of IDE complaints
> around missing types and the like, and those will likely persist until the
> code area that you're working on is compiled because the types won't exist
> until the sources are generated at pre-processing stage. Not sure if that
> was the issue you were having, but if so hopefully this helps.
>
> Thanks,
> Evan
>
> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <ma...@neo4j.com>
> wrote:
>
>> Thanks Chamikara but I'm quite familiar with the Beam API and the
>> contribution guide did not answer my questions.
>>
>> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> If you haven't already, going through Beam contribution guide and varils
>>> links from there might help: https://beam.apache.org/contribute/
>>> Regarding developing I/O connectors, please see the guide here:
>>> https://beam.apache.org/documentation/io/developing-io-overview/
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
>>> aromanenko.dev@gmail.com> wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com>
>>>> wrote:
>>>>
>>>> 1) Setting up my Beam development
>>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>>>> configurations.
>>>>
>>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>>>> seems outdated documentation
>>>> <https://github.com/google/auto/blob/master/value/userguide/index.md>?
>>>>
>>>>
>>>> Could you elaborate more what is wrong with 1) and 2) ?
>>>>
>>>
>>>> 3) Since I'm obviously planning to generate a PR at the end of this
>>>> exercise: what is the suggested code format for Java in the Beam project?
>>>>
>>>>
>>>> Please, run this command before committing your changes:
>>>>
>>>> ./gradlew spotlessApply && ./gradlew
>>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>>>> spotbugsMain compileJava compileTestJava
>>>>
>>>> To save a time, run it only against a package where you did the changes.
>>>>
>>>> —
>>>> Alexey
>>>>
>>>>
>>
>> --
>> Neo4j Chief Solutions Architect
>> *✉ *matt.casters@neo4j.com
>>
>>
>>
>>
Re: IO Connector
Posted by Matt Casters <ma...@neo4j.com>.
I promised Sachin I would provide the feedback and so I did. As mentioned
before I had not used gradle before Beam so it's also quite possible that
this is a problem between keyboard and chair and that I simply
misunderstood something early on in the (quite lengthy) instruction set.
That being said, I *was* surprised that Ubuntu 20.04.3 LTS had such an old
version of gradle installed and when I updated it everything started to
work as advertised.
Thanks for caring!
Matt
Re: IO Connector
Posted by Luke Cwik <lc...@google.com>.
Sorry to hear about the trouble with getting your workspace setup.
I was under the impression that if you were using ./gradlew (the gradle
wrapper) it would download the version of Gradle that is used within the
project automatically and wouldn't use the installed version. Using
./gradle would use the installed version on your machine.
On Thu, Nov 4, 2021 at 9:29 AM Matt Casters <ma...@neotechnology.com>
wrote:
> Dear Beams,
>
> I've made progress with the Neo4jIO source/sink implementation and left a
> trail in BEAM-1857
> <https://issues.apache.org/jira/browse/BEAM-1857?focusedCommentId=17436020&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17436020>
> .
> I'll continue to test a bit more with more data to test some edge cases
> and then I'll generate a PR so you can review it.
>
> TLDR: I promised you and some folks at Google a review of my struggles
> to get this IO developed so below is that.
>
> It looks like most of my initial de-railing came from an old gradle
> version which came by default on my Ubuntu 20.04.3 LTS environment.
> The Beam Contribute <https://beam.apache.org/contribute/> page, while
> quite lengthy, seems to be accurate but perhaps it would be prudent to also
> mention a minimal gradle version just to be on the safe side.
>
> On a side note I would advise developers to first fork apache/beam instead
> of cloning it directly so that it's easier to generate PRs later on.
> Here's an example
> <http://hop.apache.org/dev-manual/latest/setup-dev-environment.html> on
> how we approached this in Apache Hop.
> Perhaps it can remove unnecessary hurdles later on.
>
> The next point where things went awry is that the IntelliJ IDEA setup
> information is a bit outdated for recent IDEA versions. (I'm on 2021.2
> myself)
> When trying it manually by following the IntelliJ IDE instructions
> <https://cwiki.apache.org/confluence/display/BEAM/Using+IntelliJ+IDE>
> things seem to go wrong, especially wrt to Gradle and Google AutoValue
> configurations.
> The feedback from this thread was very useful to get past this, more in
> particular this command:
>
> *./gradlew idea*
>
> A very useful command for sure except that as a non-gradle expert I
> couldn't for the life of me add the new sdks/java/io/neo4j module in IDEA
> after a successful import of the whole project. In the end I started from
> a clean slate and added the directory with a build.gradle file (outside of
> IntelliJ) and added *include(":sdks:java:io:neo4j")* in
> gradle.settings.kts. After that I ran *./gradlew idea* again which worked
> out. It would be great if there was an easier way but I was happy enough to
> find a solution.
>
> AutoValue itself is cool from a technology viewpoint and once it got
> properly integrated in IntelliJ IDEA it worked well enough.
> That being said, it has its own learning curve and set of peculiarities.
> Particularly the mapping between the getters/setters, the with* methods and
> the actual code in the DoFn classes is rather tedious and making any
> changes to data types (oops I used a Long instead of an Integer) becomes a
> lengthy exercise. In general I would have the tendency to spin off a
> number of these classes but AutoValue only seems to support static inlined
> classes so I fear that you tend to end up with monolithic classes with in
> general low code density. I don't mind but I have my doubts that the code
> is actually easier to maintain this way.
>
> Adding more comments would be nitpicking since on the whole I've liked
> developing this IO ;-)
>
> Thanks again for all the help!
>
> Cheers,
> Matt
>
>
> On Tue, Oct 19, 2021 at 9:20 AM Matt Casters <ma...@neo4j.com>
> wrote:
>
>> Thanks Brian,
>>
>> I couldn't so I noticed with "git status" that something had changed some
>> gradle files. Probably an earlier attempt with the older gradle version or
>> IDEA. Getting everything back in sync with upstream master all of a sudden
>> made everything come back to life. Thanks for sticking with me!
>> When I'm done I'll document the whole process.
>>
>> Cheers,
>> Matt
>>
>> On Mon, Oct 18, 2021 at 9:10 PM Brian Hulette <bh...@google.com>
>> wrote:
>>
>>> Can you run "./gradlew sdks:java:core:compileJava" or "./gradlew
>>> sdks:java:core:test"? I'm not sure why gradle is failing to find the idea
>>> task for you - could you share the output of "./gradlew tasks"?
>>>
>>> On Mon, Oct 18, 2021 at 1:19 AM Matt Casters <
>>> matt.casters@neotechnology.com> wrote:
>>>
>>>> Thanks a lot for the advice given last week.
>>>> Just to circle back: I've updated gradle to a recent version which
>>>> appears to be 7.1.1. ... to no avail.
>>>>
>>>> tasks like:
>>>>
>>>> ./gradlew idea
>>>>
>>>> result simply in
>>>>
>>>> Task 'idea' not found in root project 'beam'.
>>>>
>>>> The same goes for the other suggestions.
>>>>
>>>> As for Google Auto: is this project still maintained? The docs and so
>>>> on seem to be getting quite old.
>>>> The annotation processor in the latest IntelliJ doesn't seem to get
>>>> picked up even if you configure it manually in the settings.
>>>> So I'll skip that one for now.
>>>>
>>>> Are there any build instructions I can follow for Beam to at least try
>>>> to build the Java SDK and go from there?
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>> On Tue, Oct 12, 2021 at 9:58 PM Evan Galpin <ev...@gmail.com>
>>>> wrote:
>>>>
>>>>> @Matt have you tried any of the "IDE Tasks" available through gradle?
>>>>> "./gradlew tasks" from beam top-level will list available tasks, and the
>>>>> IDE Tasks subsection includes tasks specific to trying to bootstrap or
>>>>> clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea"
>>>>> should set up the project files for use in Intellij. There's also
>>>>> "./gradlew cleanIdea" which may be helpful to you.
>>>>>
>>>>> With respect to Google Auto, I've experienced plenty of IDE complaints
>>>>> around missing types and the like, and those will likely persist until the
>>>>> code area that you're working on is compiled because the types won't exist
>>>>> until the sources are generated at pre-processing stage. Not sure if that
>>>>> was the issue you were having, but if so hopefully this helps.
>>>>>
>>>>> Thanks,
>>>>> Evan
>>>>>
>>>>> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <ma...@neo4j.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks Chamikara but I'm quite familiar with the Beam API and the
>>>>>> contribution guide did not answer my questions.
>>>>>>
>>>>>> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <
>>>>>> chamikara@google.com> wrote:
>>>>>>
>>>>>>> If you haven't already, going through Beam contribution guide and
>>>>>>> varils links from there might help:
>>>>>>> https://beam.apache.org/contribute/
>>>>>>> Regarding developing I/O connectors, please see the guide here:
>>>>>>> https://beam.apache.org/documentation/io/developing-io-overview/
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Cham
>>>>>>>
>>>>>>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
>>>>>>> aromanenko.dev@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Matt,
>>>>>>>>
>>>>>>>> On 12 Oct 2021, at 10:02, Matt Casters <
>>>>>>>> matt.casters@neotechnology.com> wrote:
>>>>>>>>
>>>>>>>> 1) Setting up my Beam development
>>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>>>>>>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>>>>>>>> configurations.
>>>>>>>>
>>>>>>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>>>>>>>> seems outdated documentation
>>>>>>>> <https://github.com/google/auto/blob/master/value/userguide/index.md>
>>>>>>>> ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Could you elaborate more what is wrong with 1) and 2) ?
>>>>>>>>
>>>>>>>
>>>>>>>> 3) Since I'm obviously planning to generate a PR at the end of this
>>>>>>>> exercise: what is the suggested code format for Java in the Beam project?
>>>>>>>>
>>>>>>>>
>>>>>>>> Please, run this command before committing your changes:
>>>>>>>>
>>>>>>>> ./gradlew spotlessApply && ./gradlew
>>>>>>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>>>>>>>> spotbugsMain compileJava compileTestJava
>>>>>>>>
>>>>>>>> To save a time, run it only against a package where you did the
>>>>>>>> changes.
>>>>>>>>
>>>>>>>> —
>>>>>>>> Alexey
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Neo4j Chief Solutions Architect
>>>>>> *✉ *matt.casters@neo4j.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>
>> --
>> Neo4j Chief Solutions Architect
>> *✉ *matt.casters@neo4j.com
>>
>>
>>
>>
Re: IO Connector
Posted by Matt Casters <ma...@neotechnology.com>.
Dear Beams,
I've made progress with the Neo4jIO source/sink implementation and left a
trail in BEAM-1857
<https://issues.apache.org/jira/browse/BEAM-1857?focusedCommentId=17436020&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17436020>
.
I'll continue to test a bit more with more data to test some edge cases and
then I'll generate a PR so you can review it.
TLDR: I promised you and some folks at Google a review of my struggles
to get this IO developed so below is that.
It looks like most of my initial de-railing came from an old gradle version
which came by default on my Ubuntu 20.04.3 LTS environment.
The Beam Contribute <https://beam.apache.org/contribute/> page, while quite
lengthy, seems to be accurate but perhaps it would be prudent to also
mention a minimal gradle version just to be on the safe side.
On a side note I would advise developers to first fork apache/beam instead
of cloning it directly so that it's easier to generate PRs later on.
Here's an example
<http://hop.apache.org/dev-manual/latest/setup-dev-environment.html> on how
we approached this in Apache Hop.
Perhaps it can remove unnecessary hurdles later on.
The next point where things went awry is that the IntelliJ IDEA setup
information is a bit outdated for recent IDEA versions. (I'm on 2021.2
myself)
When trying it manually by following the IntelliJ IDE instructions
<https://cwiki.apache.org/confluence/display/BEAM/Using+IntelliJ+IDE>
things seem to go wrong, especially wrt to Gradle and Google AutoValue
configurations.
The feedback from this thread was very useful to get past this, more in
particular this command:
*./gradlew idea*
A very useful command for sure except that as a non-gradle expert I
couldn't for the life of me add the new sdks/java/io/neo4j module in IDEA
after a successful import of the whole project. In the end I started from
a clean slate and added the directory with a build.gradle file (outside of
IntelliJ) and added *include(":sdks:java:io:neo4j")* in
gradle.settings.kts. After that I ran *./gradlew idea* again which worked
out. It would be great if there was an easier way but I was happy enough to
find a solution.
AutoValue itself is cool from a technology viewpoint and once it got
properly integrated in IntelliJ IDEA it worked well enough.
That being said, it has its own learning curve and set of peculiarities.
Particularly the mapping between the getters/setters, the with* methods and
the actual code in the DoFn classes is rather tedious and making any
changes to data types (oops I used a Long instead of an Integer) becomes a
lengthy exercise. In general I would have the tendency to spin off a
number of these classes but AutoValue only seems to support static inlined
classes so I fear that you tend to end up with monolithic classes with in
general low code density. I don't mind but I have my doubts that the code
is actually easier to maintain this way.
Adding more comments would be nitpicking since on the whole I've liked
developing this IO ;-)
Thanks again for all the help!
Cheers,
Matt
On Tue, Oct 19, 2021 at 9:20 AM Matt Casters <ma...@neo4j.com> wrote:
> Thanks Brian,
>
> I couldn't so I noticed with "git status" that something had changed some
> gradle files. Probably an earlier attempt with the older gradle version or
> IDEA. Getting everything back in sync with upstream master all of a sudden
> made everything come back to life. Thanks for sticking with me!
> When I'm done I'll document the whole process.
>
> Cheers,
> Matt
>
> On Mon, Oct 18, 2021 at 9:10 PM Brian Hulette <bh...@google.com> wrote:
>
>> Can you run "./gradlew sdks:java:core:compileJava" or "./gradlew
>> sdks:java:core:test"? I'm not sure why gradle is failing to find the idea
>> task for you - could you share the output of "./gradlew tasks"?
>>
>> On Mon, Oct 18, 2021 at 1:19 AM Matt Casters <
>> matt.casters@neotechnology.com> wrote:
>>
>>> Thanks a lot for the advice given last week.
>>> Just to circle back: I've updated gradle to a recent version which
>>> appears to be 7.1.1. ... to no avail.
>>>
>>> tasks like:
>>>
>>> ./gradlew idea
>>>
>>> result simply in
>>>
>>> Task 'idea' not found in root project 'beam'.
>>>
>>> The same goes for the other suggestions.
>>>
>>> As for Google Auto: is this project still maintained? The docs and so on
>>> seem to be getting quite old.
>>> The annotation processor in the latest IntelliJ doesn't seem to get
>>> picked up even if you configure it manually in the settings.
>>> So I'll skip that one for now.
>>>
>>> Are there any build instructions I can follow for Beam to at least try
>>> to build the Java SDK and go from there?
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>> On Tue, Oct 12, 2021 at 9:58 PM Evan Galpin <ev...@gmail.com>
>>> wrote:
>>>
>>>> @Matt have you tried any of the "IDE Tasks" available through gradle?
>>>> "./gradlew tasks" from beam top-level will list available tasks, and the
>>>> IDE Tasks subsection includes tasks specific to trying to bootstrap or
>>>> clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea"
>>>> should set up the project files for use in Intellij. There's also
>>>> "./gradlew cleanIdea" which may be helpful to you.
>>>>
>>>> With respect to Google Auto, I've experienced plenty of IDE complaints
>>>> around missing types and the like, and those will likely persist until the
>>>> code area that you're working on is compiled because the types won't exist
>>>> until the sources are generated at pre-processing stage. Not sure if that
>>>> was the issue you were having, but if so hopefully this helps.
>>>>
>>>> Thanks,
>>>> Evan
>>>>
>>>> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <ma...@neo4j.com>
>>>> wrote:
>>>>
>>>>> Thanks Chamikara but I'm quite familiar with the Beam API and the
>>>>> contribution guide did not answer my questions.
>>>>>
>>>>> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <
>>>>> chamikara@google.com> wrote:
>>>>>
>>>>>> If you haven't already, going through Beam contribution guide and
>>>>>> varils links from there might help:
>>>>>> https://beam.apache.org/contribute/
>>>>>> Regarding developing I/O connectors, please see the guide here:
>>>>>> https://beam.apache.org/documentation/io/developing-io-overview/
>>>>>>
>>>>>> Thanks,
>>>>>> Cham
>>>>>>
>>>>>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
>>>>>> aromanenko.dev@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Matt,
>>>>>>>
>>>>>>> On 12 Oct 2021, at 10:02, Matt Casters <
>>>>>>> matt.casters@neotechnology.com> wrote:
>>>>>>>
>>>>>>> 1) Setting up my Beam development
>>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>>>>>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>>>>>>> configurations.
>>>>>>>
>>>>>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>>>>>>> seems outdated documentation
>>>>>>> <https://github.com/google/auto/blob/master/value/userguide/index.md>
>>>>>>> ?
>>>>>>>
>>>>>>>
>>>>>>> Could you elaborate more what is wrong with 1) and 2) ?
>>>>>>>
>>>>>>
>>>>>>> 3) Since I'm obviously planning to generate a PR at the end of this
>>>>>>> exercise: what is the suggested code format for Java in the Beam project?
>>>>>>>
>>>>>>>
>>>>>>> Please, run this command before committing your changes:
>>>>>>>
>>>>>>> ./gradlew spotlessApply && ./gradlew
>>>>>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>>>>>>> spotbugsMain compileJava compileTestJava
>>>>>>>
>>>>>>> To save a time, run it only against a package where you did the
>>>>>>> changes.
>>>>>>>
>>>>>>> —
>>>>>>> Alexey
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Neo4j Chief Solutions Architect
>>>>> *✉ *matt.casters@neo4j.com
>>>>>
>>>>>
>>>>>
>>>>>
>
> --
> Neo4j Chief Solutions Architect
> *✉ *matt.casters@neo4j.com
>
>
>
>
Re: IO Connector
Posted by Matt Casters <ma...@neo4j.com>.
Thanks Brian,
I couldn't so I noticed with "git status" that something had changed some
gradle files. Probably an earlier attempt with the older gradle version or
IDEA. Getting everything back in sync with upstream master all of a sudden
made everything come back to life. Thanks for sticking with me!
When I'm done I'll document the whole process.
Cheers,
Matt
On Mon, Oct 18, 2021 at 9:10 PM Brian Hulette <bh...@google.com> wrote:
> Can you run "./gradlew sdks:java:core:compileJava" or "./gradlew
> sdks:java:core:test"? I'm not sure why gradle is failing to find the idea
> task for you - could you share the output of "./gradlew tasks"?
>
> On Mon, Oct 18, 2021 at 1:19 AM Matt Casters <
> matt.casters@neotechnology.com> wrote:
>
>> Thanks a lot for the advice given last week.
>> Just to circle back: I've updated gradle to a recent version which
>> appears to be 7.1.1. ... to no avail.
>>
>> tasks like:
>>
>> ./gradlew idea
>>
>> result simply in
>>
>> Task 'idea' not found in root project 'beam'.
>>
>> The same goes for the other suggestions.
>>
>> As for Google Auto: is this project still maintained? The docs and so on
>> seem to be getting quite old.
>> The annotation processor in the latest IntelliJ doesn't seem to get
>> picked up even if you configure it manually in the settings.
>> So I'll skip that one for now.
>>
>> Are there any build instructions I can follow for Beam to at least try to
>> build the Java SDK and go from there?
>>
>> Thanks,
>>
>> Matt
>>
>>
>> On Tue, Oct 12, 2021 at 9:58 PM Evan Galpin <ev...@gmail.com>
>> wrote:
>>
>>> @Matt have you tried any of the "IDE Tasks" available through gradle?
>>> "./gradlew tasks" from beam top-level will list available tasks, and the
>>> IDE Tasks subsection includes tasks specific to trying to bootstrap or
>>> clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea"
>>> should set up the project files for use in Intellij. There's also
>>> "./gradlew cleanIdea" which may be helpful to you.
>>>
>>> With respect to Google Auto, I've experienced plenty of IDE complaints
>>> around missing types and the like, and those will likely persist until the
>>> code area that you're working on is compiled because the types won't exist
>>> until the sources are generated at pre-processing stage. Not sure if that
>>> was the issue you were having, but if so hopefully this helps.
>>>
>>> Thanks,
>>> Evan
>>>
>>> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <ma...@neo4j.com>
>>> wrote:
>>>
>>>> Thanks Chamikara but I'm quite familiar with the Beam API and the
>>>> contribution guide did not answer my questions.
>>>>
>>>> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <
>>>> chamikara@google.com> wrote:
>>>>
>>>>> If you haven't already, going through Beam contribution guide and
>>>>> varils links from there might help:
>>>>> https://beam.apache.org/contribute/
>>>>> Regarding developing I/O connectors, please see the guide here:
>>>>> https://beam.apache.org/documentation/io/developing-io-overview/
>>>>>
>>>>> Thanks,
>>>>> Cham
>>>>>
>>>>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
>>>>> aromanenko.dev@gmail.com> wrote:
>>>>>
>>>>>> Hi Matt,
>>>>>>
>>>>>> On 12 Oct 2021, at 10:02, Matt Casters <
>>>>>> matt.casters@neotechnology.com> wrote:
>>>>>>
>>>>>> 1) Setting up my Beam development
>>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>>>>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>>>>>> configurations.
>>>>>>
>>>>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>>>>>> seems outdated documentation
>>>>>> <https://github.com/google/auto/blob/master/value/userguide/index.md>
>>>>>> ?
>>>>>>
>>>>>>
>>>>>> Could you elaborate more what is wrong with 1) and 2) ?
>>>>>>
>>>>>
>>>>>> 3) Since I'm obviously planning to generate a PR at the end of this
>>>>>> exercise: what is the suggested code format for Java in the Beam project?
>>>>>>
>>>>>>
>>>>>> Please, run this command before committing your changes:
>>>>>>
>>>>>> ./gradlew spotlessApply && ./gradlew
>>>>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>>>>>> spotbugsMain compileJava compileTestJava
>>>>>>
>>>>>> To save a time, run it only against a package where you did the
>>>>>> changes.
>>>>>>
>>>>>> —
>>>>>> Alexey
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Neo4j Chief Solutions Architect
>>>> *✉ *matt.casters@neo4j.com
>>>>
>>>>
>>>>
>>>>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: IO Connector
Posted by Brian Hulette <bh...@google.com>.
Can you run "./gradlew sdks:java:core:compileJava" or "./gradlew
sdks:java:core:test"? I'm not sure why gradle is failing to find the idea
task for you - could you share the output of "./gradlew tasks"?
On Mon, Oct 18, 2021 at 1:19 AM Matt Casters <ma...@neotechnology.com>
wrote:
> Thanks a lot for the advice given last week.
> Just to circle back: I've updated gradle to a recent version which appears
> to be 7.1.1. ... to no avail.
>
> tasks like:
>
> ./gradlew idea
>
> result simply in
>
> Task 'idea' not found in root project 'beam'.
>
> The same goes for the other suggestions.
>
> As for Google Auto: is this project still maintained? The docs and so on
> seem to be getting quite old.
> The annotation processor in the latest IntelliJ doesn't seem to get picked
> up even if you configure it manually in the settings.
> So I'll skip that one for now.
>
> Are there any build instructions I can follow for Beam to at least try to
> build the Java SDK and go from there?
>
> Thanks,
>
> Matt
>
>
> On Tue, Oct 12, 2021 at 9:58 PM Evan Galpin <ev...@gmail.com> wrote:
>
>> @Matt have you tried any of the "IDE Tasks" available through gradle?
>> "./gradlew tasks" from beam top-level will list available tasks, and the
>> IDE Tasks subsection includes tasks specific to trying to bootstrap or
>> clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea"
>> should set up the project files for use in Intellij. There's also
>> "./gradlew cleanIdea" which may be helpful to you.
>>
>> With respect to Google Auto, I've experienced plenty of IDE complaints
>> around missing types and the like, and those will likely persist until the
>> code area that you're working on is compiled because the types won't exist
>> until the sources are generated at pre-processing stage. Not sure if that
>> was the issue you were having, but if so hopefully this helps.
>>
>> Thanks,
>> Evan
>>
>> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <ma...@neo4j.com>
>> wrote:
>>
>>> Thanks Chamikara but I'm quite familiar with the Beam API and the
>>> contribution guide did not answer my questions.
>>>
>>> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <ch...@google.com>
>>> wrote:
>>>
>>>> If you haven't already, going through Beam contribution guide and
>>>> varils links from there might help: https://beam.apache.org/contribute/
>>>> Regarding developing I/O connectors, please see the guide here:
>>>> https://beam.apache.org/documentation/io/developing-io-overview/
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
>>>> aromanenko.dev@gmail.com> wrote:
>>>>
>>>>> Hi Matt,
>>>>>
>>>>> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com>
>>>>> wrote:
>>>>>
>>>>> 1) Setting up my Beam development
>>>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>>>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>>>>> configurations.
>>>>>
>>>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>>>>> seems outdated documentation
>>>>> <https://github.com/google/auto/blob/master/value/userguide/index.md>?
>>>>>
>>>>>
>>>>> Could you elaborate more what is wrong with 1) and 2) ?
>>>>>
>>>>
>>>>> 3) Since I'm obviously planning to generate a PR at the end of this
>>>>> exercise: what is the suggested code format for Java in the Beam project?
>>>>>
>>>>>
>>>>> Please, run this command before committing your changes:
>>>>>
>>>>> ./gradlew spotlessApply && ./gradlew
>>>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>>>>> spotbugsMain compileJava compileTestJava
>>>>>
>>>>> To save a time, run it only against a package where you did the
>>>>> changes.
>>>>>
>>>>> —
>>>>> Alexey
>>>>>
>>>>>
>>>
>>> --
>>> Neo4j Chief Solutions Architect
>>> *✉ *matt.casters@neo4j.com
>>>
>>>
>>>
>>>
Re: IO Connector
Posted by Matt Casters <ma...@neotechnology.com>.
Thanks a lot for the advice given last week.
Just to circle back: I've updated gradle to a recent version which appears
to be 7.1.1. ... to no avail.
tasks like:
./gradlew idea
result simply in
Task 'idea' not found in root project 'beam'.
The same goes for the other suggestions.
As for Google Auto: is this project still maintained? The docs and so on
seem to be getting quite old.
The annotation processor in the latest IntelliJ doesn't seem to get picked
up even if you configure it manually in the settings.
So I'll skip that one for now.
Are there any build instructions I can follow for Beam to at least try to
build the Java SDK and go from there?
Thanks,
Matt
On Tue, Oct 12, 2021 at 9:58 PM Evan Galpin <ev...@gmail.com> wrote:
> @Matt have you tried any of the "IDE Tasks" available through gradle?
> "./gradlew tasks" from beam top-level will list available tasks, and the
> IDE Tasks subsection includes tasks specific to trying to bootstrap or
> clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea"
> should set up the project files for use in Intellij. There's also
> "./gradlew cleanIdea" which may be helpful to you.
>
> With respect to Google Auto, I've experienced plenty of IDE complaints
> around missing types and the like, and those will likely persist until the
> code area that you're working on is compiled because the types won't exist
> until the sources are generated at pre-processing stage. Not sure if that
> was the issue you were having, but if so hopefully this helps.
>
> Thanks,
> Evan
>
> On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <ma...@neo4j.com>
> wrote:
>
>> Thanks Chamikara but I'm quite familiar with the Beam API and the
>> contribution guide did not answer my questions.
>>
>> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> If you haven't already, going through Beam contribution guide and varils
>>> links from there might help: https://beam.apache.org/contribute/
>>> Regarding developing I/O connectors, please see the guide here:
>>> https://beam.apache.org/documentation/io/developing-io-overview/
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
>>> aromanenko.dev@gmail.com> wrote:
>>>
>>>> Hi Matt,
>>>>
>>>> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com>
>>>> wrote:
>>>>
>>>> 1) Setting up my Beam development
>>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>>>> configurations.
>>>>
>>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>>>> seems outdated documentation
>>>> <https://github.com/google/auto/blob/master/value/userguide/index.md>?
>>>>
>>>>
>>>> Could you elaborate more what is wrong with 1) and 2) ?
>>>>
>>>
>>>> 3) Since I'm obviously planning to generate a PR at the end of this
>>>> exercise: what is the suggested code format for Java in the Beam project?
>>>>
>>>>
>>>> Please, run this command before committing your changes:
>>>>
>>>> ./gradlew spotlessApply && ./gradlew
>>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>>>> spotbugsMain compileJava compileTestJava
>>>>
>>>> To save a time, run it only against a package where you did the changes.
>>>>
>>>> —
>>>> Alexey
>>>>
>>>>
>>
>> --
>> Neo4j Chief Solutions Architect
>> *✉ *matt.casters@neo4j.com
>>
>>
>>
>>
Re: IO Connector
Posted by Evan Galpin <ev...@gmail.com>.
@Matt have you tried any of the "IDE Tasks" available through gradle?
"./gradlew tasks" from beam top-level will list available tasks, and the
IDE Tasks subsection includes tasks specific to trying to bootstrap or
clean up beam project in either Eclipse or Intellij. Ex. "./gradlew idea"
should set up the project files for use in Intellij. There's also
"./gradlew cleanIdea" which may be helpful to you.
With respect to Google Auto, I've experienced plenty of IDE complaints
around missing types and the like, and those will likely persist until the
code area that you're working on is compiled because the types won't exist
until the sources are generated at pre-processing stage. Not sure if that
was the issue you were having, but if so hopefully this helps.
Thanks,
Evan
On Tue, Oct 12, 2021 at 2:51 PM Matt Casters <ma...@neo4j.com> wrote:
> Thanks Chamikara but I'm quite familiar with the Beam API and the
> contribution guide did not answer my questions.
>
> On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> If you haven't already, going through Beam contribution guide and varils
>> links from there might help: https://beam.apache.org/contribute/
>> Regarding developing I/O connectors, please see the guide here:
>> https://beam.apache.org/documentation/io/developing-io-overview/
>>
>> Thanks,
>> Cham
>>
>> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <
>> aromanenko.dev@gmail.com> wrote:
>>
>>> Hi Matt,
>>>
>>> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com>
>>> wrote:
>>>
>>> 1) Setting up my Beam development
>>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>>> configurations.
>>>
>>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what
>>> seems outdated documentation
>>> <https://github.com/google/auto/blob/master/value/userguide/index.md>?
>>>
>>>
>>> Could you elaborate more what is wrong with 1) and 2) ?
>>>
>>
>>> 3) Since I'm obviously planning to generate a PR at the end of this
>>> exercise: what is the suggested code format for Java in the Beam project?
>>>
>>>
>>> Please, run this command before committing your changes:
>>>
>>> ./gradlew spotlessApply && ./gradlew
>>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>>> spotbugsMain compileJava compileTestJava
>>>
>>> To save a time, run it only against a package where you did the changes.
>>>
>>> —
>>> Alexey
>>>
>>>
>
> --
> Neo4j Chief Solutions Architect
> *✉ *matt.casters@neo4j.com
>
>
>
>
Re: IO Connector
Posted by Matt Casters <ma...@neo4j.com>.
Thanks Chamikara but I'm quite familiar with the Beam API and the
contribution guide did not answer my questions.
On Tue, Oct 12, 2021 at 8:49 PM Chamikara Jayalath <ch...@google.com>
wrote:
> If you haven't already, going through Beam contribution guide and varils
> links from there might help: https://beam.apache.org/contribute/
> Regarding developing I/O connectors, please see the guide here:
> https://beam.apache.org/documentation/io/developing-io-overview/
>
> Thanks,
> Cham
>
> On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <ar...@gmail.com>
> wrote:
>
>> Hi Matt,
>>
>> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com>
>> wrote:
>>
>> 1) Setting up my Beam development
>> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
>> for IDEA 2021.2 is something that's going wrong, probably around Gradle
>> configurations.
>>
>> 2) I can't get Google Auto to work in my IDE (IDEA) because of what seems
>> outdated documentation
>> <https://github.com/google/auto/blob/master/value/userguide/index.md>?
>>
>>
>> Could you elaborate more what is wrong with 1) and 2) ?
>>
>
>> 3) Since I'm obviously planning to generate a PR at the end of this
>> exercise: what is the suggested code format for Java in the Beam project?
>>
>>
>> Please, run this command before committing your changes:
>>
>> ./gradlew spotlessApply && ./gradlew
>> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
>> spotbugsMain compileJava compileTestJava
>>
>> To save a time, run it only against a package where you did the changes.
>>
>> —
>> Alexey
>>
>>
--
Neo4j Chief Solutions Architect
*✉ *matt.casters@neo4j.com
Re: IO Connector
Posted by Chamikara Jayalath <ch...@google.com>.
If you haven't already, going through Beam contribution guide and varils
links from there might help: https://beam.apache.org/contribute/
Regarding developing I/O connectors, please see the guide here:
https://beam.apache.org/documentation/io/developing-io-overview/
Thanks,
Cham
On Tue, Oct 12, 2021 at 6:08 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Hi Matt,
>
> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com>
> wrote:
>
> 1) Setting up my Beam development
> <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment
> for IDEA 2021.2 is something that's going wrong, probably around Gradle
> configurations.
>
> 2) I can't get Google Auto to work in my IDE (IDEA) because of what seems
> outdated documentation
> <https://github.com/google/auto/blob/master/value/userguide/index.md>?
>
>
> Could you elaborate more what is wrong with 1) and 2) ?
>
> 3) Since I'm obviously planning to generate a PR at the end of this
> exercise: what is the suggested code format for Java in the Beam project?
>
>
> Please, run this command before committing your changes:
>
> ./gradlew spotlessApply && ./gradlew
> -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc
> spotbugsMain compileJava compileTestJava
>
> To save a time, run it only against a package where you did the changes.
>
> —
> Alexey
>
>
Re: IO Connector
Posted by Alexey Romanenko <ar...@gmail.com>.
Hi Matt,
> On 12 Oct 2021, at 10:02, Matt Casters <ma...@neotechnology.com> wrote:
>
> 1) Setting up my Beam development <https://cwiki.apache.org/confluence/display/BEAM/Set+up+IntelliJ+from+scratch> environment for IDEA 2021.2 is something that's going wrong, probably around Gradle configurations.
> 2) I can't get Google Auto to work in my IDE (IDEA) because of what seems outdated documentation <https://github.com/google/auto/blob/master/value/userguide/index.md>?
Could you elaborate more what is wrong with 1) and 2) ?
> 3) Since I'm obviously planning to generate a PR at the end of this exercise: what is the suggested code format for Java in the Beam project?
Please, run this command before committing your changes:
./gradlew spotlessApply && ./gradlew -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc spotbugsMain compileJava compileTestJava
To save a time, run it only against a package where you did the changes.
—
Alexey