You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@daffodil.apache.org by Davin Shearer <sc...@gmail.com> on 2023/06/02 19:23:56 UTC
Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Thanks for the detailed update and your patience Mike. This is
definitely not the user experience we're hoping for. Are these regressions
that have occurred since the v1.2.0 release?
I'm a big fan of compartmentalization and technologies like docker and
python's venv (virtual environment) among many others. Relying on system
installs of virtual machines (e.g. Java's JVM) and interpreters (e.g.,
Python, Perl, etc) introduces uncontrolled variables into our runtime
environment, which I believe is one of the major encumbrances that you're
experiencing and something I hope we can address as a community.
I don't think docker is the right answer here because that introduces
another system-level dependency... the docker service. I think packaging
a JVM or pulling it in as a VS Code dependency (not a system level
dependency like docker) will give us much tighter control over our runtime
environment. We'll need to be careful too about how we execute our
services on that JVM so its execution environment is properly
compartmentalized. In addition to the Daffodil debugger service, the Data
Editor part of the extension also uses a service running on the JVM which
will benefit from this compartmentalization. This also will allow us to
upgrade from Java 8 to something more recent (this will help with domain
sockets and named fifos too). Anyway, that's the direction I'd be inclined
to head in, but it's definitely worth a discussion and hopefully we can
converge on solid design. I had an interesting conversation on this topic
with GPT-4 that I think is worth sharing (see:
https://chat.openai.com/share/da2ca068-714a-4eec-8a0e-8117069b3125).
Each distinct failure that you've encountered in testing RC2 needs to have
issues associated with them on the project's GitHub so we can track, triage
and prioritize, schedule, and assign them. If you're willing to enter your
issues into the GitHub tracker, that would be greatly appreciated and
you'll be able to stay in the loop on how those issues progress and help us
in the review phase to make sure that each issue is being properly and
fully addressed.
Shane is out, but I'm sure he'll have some additional follow up and things
to try before making new pull requests.
On Wed, May 31, 2023 at 7:02 PM Mike Beckerle <mb...@apache.org> wrote:
>
> So, I have the 1.3.0 RC2 vscode extension debugger working now for PCAP w/
> ethernetIP and I can set a breakpoint in ethernetIP.dfdl.xsd and I no
> longer get an NPE.
>
> Configuring this is really far too difficult, and I have to have the
> schema working first with sbt in order to even hope to configure it.
>
> But I'll get back to that later.
>
> More basic issue is that I am unable to debug even dead-simple errors with
> it. I clobbered the magic number in the first bytes of an icmp1.cap file.
> This fails an assert at line 97 of pcap.dfdl.xsd, which is the first
> element of simple type.
>
> I step through with the debugger, and nothing displays *anything* about
> the assert failure. Execution ends. A little pop-up dialog says it wrote my
> infoset out, which it didn't. There's nothing showing that anything went
> wrong. All panels on the left are blank.
>
> If I modify the schema and make any mistake, then I get a blocking dialog
> showing the SDEs. This is basically intolerable, and is why I have to go
> back to sbt to debug this. You can't even go through the SDEs as you move
> the editor. It's a blocking dialog.
>
> So both for schema compilation, and runtime errors there simply has to be
> a non-modal display of errors, probably on the left with variables, watch,
> call-stack, and breakpoints. But a quasi-edit-buffer containing them would
> also be fine, or even just output them to the terminal window.
>
> The rest of this email is about the hoops I had to jump through to get
> this to work at all. Arguably, these issues are about working with a
> multi-component schema, and we could define our objectives to simpler
> schemas than this, but that puts it in the realm of toy-only schemas. The
> example is just PCAP + ethernetIP, which is not an extreme example.
>
> The existing shell environment of the terminal definitely causes problems.
> But just exiting that so that the debugger starts a new terminal doesn't
> fix it, as that inherits the environment from where vscode was started. It
> starts a new login shell unless you take steps to prevent that.
>
> A big trip-up in usability is that the wizard that creates the
> launch.json, doesn't actually let you populate the daffodilDebugClaspath.
> You have to edit that by hand, and nothing verifies it, so any typo and you
> get tedious failures that are hard to figure out. You just get an SDE - if
> you are lucky - via a blocking dialog.
>
> The wizard should help you put directories and corresponding jars on the
> daffodilDebugClasspath, and it should not let you put a non-existing
> directory or jar file on it. Ideally, at launch time the
> daffodilDebugClasspath should be checked, just in case something on it no
> longer exists.
>
> If the debugger happens to find a file on the daffodilDebugClasspath in a
> jar, vs. in a regular directory (maybe you have a typo in the regular
> directory path), so it passes that and hits the jar, then setting a
> breakpoint causes a NPE. Something requires a "hierarchical URL" which a
> jar-file URL is not. This is the bug that really has to be fixed so that
> you can set breakpoints regardless of whether the source DFDL schema file
> was in a jar or not.
>
> Note: at Owl we have DFDL schemas where the starting root element of the
> schema isn't even part of the schema, it's part of a component schema that
> is in a jar on the classpath. This happens when you have an
> envelope-payload idiom. The schema that combines the envelope and payload
> together contains only the glue part that connects the two together. The
> root element is the envelope's root element, and that schema is in a jar
> file.
>
> I did not see any feature in launch.json for specifying the name/namespace
> of the root element. There is just the "program" element, which must be to
> a file in the current schema. That needs to be able to be specified just as
> element name + namespace.
>
> The jars have to be on the daffodilDebugClasspath because of the plugins
> that daffodil must find in them. That's understandable, the problem is
> having to also have a regular file location for any schema you want to be
> able to put a breakpoint in and it must be before the corresponding jar.
>
> That all said, here are the scripts and settings I used to get it to work:
>
> First, I have this script I use to launch vscode in a minimal/empty
> environment. This technique is useful to
> determine if you are depending on the environment in any way:
>
> #! /bin/bash
> if [ "$1" = "emptyEnv" ] # is it the special first arg
> then
> shift # discard special first arg, and continue with the rest of the
> script
> else
> CMD=$0
> exec -c $CMD "emptyEnv" $* # exec with an empty environment, special
> first arg
> fi
> # the environment is empty when we get here
> # just add env vars we need to get vscode to work
> export LANG=en_US.UTF-8
> export HOME=/home/mbeckerle
> export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 # java 11 required
> export PATH=$JAVA_HOME/bin:$PATH
> export DISPLAY=:0
> export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus # vscode
> spits lots of errors without this
> exec /usr/bin/code
> # end
>
> Then, within ~/.config/Code/User/settings.json, I had to create these
> settings for terminal:
>
> "terminal.integrated.inheritEnv": false,
> "terminal.integrated.profiles.linux": {
> "Custom Init": {
> "path": "/bin/bash",
> "args": [
> "--norc"
> ]
> }
> },
> "terminal.integrated.defaultProfile.linux": "Custom Init"
>
> That instructs vscode not to create a login shell for the terminal. The
> --norc option means it does not run the .bashrc file so that the java
> version specified in the launch script sticks.
>
> I am not sure the inheritEnv setting actually does anything. That alone
> sounds like the right thing, but that alone didn't solve the problem.
> You still have everything in the login shell environment where vscode was
> launched. Hence the need for the environment-emptying launch script.
>
> With those two things, a terminal created by vscode does not inherit
> everything my typical shell has. For example there is no daffodil on the
> path, no DAFFODIL_CLASSPATH env var, etc.
>
> Given the above if I just create a terminal with Terminal > new Terminal:
> The environment is relatively simple:
>
> bash-5.0$ env
> COLORTERM=truecolor
> TERM_PROGRAM_VERSION=1.78.2
> JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
> NO_AT_BRIDGE=1
> PWD=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP
> VSCODE_GIT_ASKPASS_NODE=/usr/share/code/code
> HOME=/home/mbeckerle
> LANG=en_US.UTF-8
> GIT_ASKPASS=/usr/share/code/resources/app/extensions/git/dist/askpass.sh
> CHROME_DESKTOP=code-url-handler.desktop
> VSCODE_GIT_ASKPASS_EXTRA_ARGS=--ms-enable-electron-run-as-node
> TERM=xterm-256color
> VSCODE_GIT_IPC_HANDLE=/tmp/vscode-git-56a269c0f5.sock
> DISPLAY=:0
> SHLVL=1
>
> VSCODE_GIT_ASKPASS_MAIN=/usr/share/code/resources/app/extensions/git/dist/askpass-main.js
> GDK_BACKEND=x11
>
> PATH=/usr/lib/jvm/java-11-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> ORIGINAL_XDG_CURRENT_DESKTOP=undefined
> DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> TERM_PROGRAM=vscode
> _=/usr/bin/env
>
> If I run daffodil, then c-C in the terminal it creates, then the
> environment has this added environment variable only.
>
>
> DAFFODIL_DEBUG_CLASSPATH=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/../ethernetIP/src/main/resources:/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/lib_managed/jars/com.owlcyberdefense/dfdl-ethernetip/dfdl-ethernetip-1.2.0.jar
>
> This works. My launch.json is attached.
>
>
>
>
> On Fri, May 26, 2023 at 2:06 PM Shane Dell <sh...@apache.org> wrote:
>
>> Mike,
>>
>> The VSCode Extension should be using the daffodil command that is bundled
>> with the
>> extension. The issue might still be with that shell env variable you have
>> for
>> DAFFODIL_CLASSPATH, but this might only be an issue with a terminal that
>> is already opened
>> with the name "daffodil-debugger". If you close that terminal, the next
>> you time you run it should
>> create a new terminal with the env set properly. If you could try this
>> and see if it works how you
>> expect, that would help us single out this issue out to an already active
>> terminal.
>>
>> Thank you,
>>
>> Shane Dell
>>
>>
>> On 2023/05/25 23:59:53 Mike Beckerle wrote:
>> > There are so many environment interactions....
>> >
>> > So I have discovered that the VSCode extension just seems to run
>> whatever
>> > daffodil command is on the path. It's not isolated at all from the shell
>> > environment.
>> >
>> > I found this out because I have a customized daffodil shell script that
>> > setups a bunch of DAFFODIL_CLASSPATH contents, and I couldn't figure out
>> > why those were part of the VSCode environment.
>> >
>> > Shouldn't vscode run exactly the daffodil bundled with it, unless you
>> > explicitly configure it not to?
>> >
>> > That's why it is finding the ethernetIP.dfdl.xsd file in the jar, not in
>> > the file system. Because my daffodil command sets up lots of jars on the
>> > DAFFODIL_CLASSPATH.
>> >
>> > It also depends on Java 11, breaks with Java 17. I also don't think this
>> > should be inherited from the environment.
>> >
>> > I really have a hard time using, or even testing the VSCode extension
>> if I
>> > have to scrub my shell environment of everything in it in order to make
>> the
>> > vscode extension work.
>> >
>> > But it is this kind of problem that caused people to invent docker
>> > containers.
>> >
>> > I am going to make a launcher script for VSCode that empties the
>> > environment, then supplies only what is absolutely necessary, with no
>> > outside environment assumptions.
>> >
>> > I will try to do this Friday if possible.
>> >
>> >
>> >
>> > On Wed, May 24, 2023 at 2:50 PM Shane Dell <sh...@apache.org>
>> wrote:
>> >
>> > > Mike,
>> > >
>> > > I am not sure why you would be seeing this. I am able to set the
>> > > breakpoint where you
>> > > mentioned without any issues. Can you try removing the daffodil-dap
>> > > folder? This should be
>> > > somewhere like:
>> > > /home/$USER/.local/share/daffodil-dap
>> > > The issue I am thinking is possibly a bug with the rc1 debugger that
>> may
>> > > not be present on rc2.
>> > > But if the folder daffodil-dap/daffodil-debugger-3.4.0-1.3.0/ already
>> > > exists from rc1 then rc2 will
>> > > not remake it. So in order to remake it you have to manually remove
>> this
>> > > folder. Let me know if
>> > > this helps.
>> > >
>> > > Thank you,
>> > >
>> > > Shane Dell
>> > >
>> > > On 2023/05/24 15:15:33 Mike Beckerle wrote:
>> > > > I have been able to continue evaluating RC2.
>> > > >
>> > > > My vote is -1
>> > > >
>> > > > My testing steps listed below. I am trying to debug PCAP. I have a
>> broken
>> > > > icmp packet (bad magic number), and also just step through a
>> correct icmp
>> > > > packet.
>> > > >
>> > > > I am trying to debug icmp.badMagicNum.cap data file.
>> > > >
>> > > > I should get a failure of an assert because the magic number is not
>> one
>> > > of
>> > > > the expected values.
>> > > >
>> > > > I'm just trying to single-step through it until it hits the error.
>> > > >
>> > > > Once we get to the MagicNumber element, it doesn't position on the
>> assert
>> > > > while evaluating that expression or even on the element carrying the
>> > > assert.
>> > > > It jumps back to the top-level xs:schema position.
>> > > >
>> > > > I continue to click single step. Watching for the error to be
>> displayed
>> > > in
>> > > > the upper left under Variables > Parse > Diagnostics > Errors which
>> is
>> > > > where I think it ought to go.
>> > > >
>> > > > At some point the failure is somehow detected as the debug session
>> ends.
>> > > A
>> > > > dialog offers to open /tmp/infoset.xml for me, which I open and it
>> is
>> > > empty
>> > > > of course.
>> > > >
>> > > > Nothing displayed the error message from the assert, or even
>> indicated
>> > > what
>> > > > went wrong at all. Not the state display in the upper left, not the
>> > > > terminal, nor debug console nor output.
>> > > >
>> > > > The upper left box with "Variables" is cleared at the end. It should
>> > > > certainly not be cleared at the end of a run when final state may
>> be of
>> > > > interest. At the start of a run, or manually, but not at the end of
>> a
>> > > run.
>> > > >
>> > > > Second attempt to use it:
>> > > >
>> > > > I create a second launch config this time for what should be a
>> successful
>> > > > parse of icmp1.cap
>> > > >
>> > > > As I single step, the position of the cursor on the schema is
>> jumping
>> > > > around all over the place back and forth between a containing
>> element and
>> > > > the child elements it contains.
>> > > > The infoset display does not get updated until long after the
>> display has
>> > > > moved on and is positioned on the next element.
>> > > >
>> > > > This jumping around gets completely out of hand once you step into
>> the
>> > > > complexType "Ethernet" which is in the ethernetIP jar file.
>> > > >
>> > > > At that point, after each child of Ethernet, it jumps back to the
>> > > Ethernet
>> > > > element in the pcap.dfdl.xsd file, and then back into the
>> > > > ethernetIP.dfdl.xsd for the next child, then back to the
>> pcap.dfdl.xsd
>> > > > file, etc.
>> > > >
>> > > > All this jumping around basically makes stepping useless. So I
>> decide to
>> > > > put a breakpoint on the Version element inside the IPv4Header
>> element
>> > > > inside ethernetIP.dfdl.xsd.
>> > > >
>> > > > Just adding the breakpoint causes the debug session to end with an
>> NPE in
>> > > > the dap setBreakpoint.
>> > > >
>> > > > ERROR o.a.d.d.d.DAPodil - unhandled error
>> > > > java.lang.NullPointerException: null
>> > > > at
>> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)
>> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
>> > > > at
>> > > > java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)
>> > > > at java.base/java.nio.file.Path.of(Path.java:147)
>> > > > at java.base/java.nio.file.Paths.get(Paths.java:69)
>> > > > at
>> > > >
>> > >
>> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @
>> > > >
>> > >
>> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at handleErrorWith @
>> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @
>> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > 2023-05-24 11:09:10,717 [io-compute-1] DEBUG o.a.d.d.d.DAPodil -
>> > > whenDone:
>> > > > completed
>> > > > java.lang.NullPointerException
>> > > > at
>> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)
>> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
>> > > > at
>> > > > java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)
>> > > > at java.base/java.nio.file.Path.of(Path.java:147)
>> > > > at java.base/java.nio.file.Paths.get(Paths.java:69)
>> > > > at
>> > > >
>> > >
>> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @
>> > > >
>> > >
>> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at handleErrorWith @
>> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > > at flatMap @
>> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929)
>> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
>> > > > at flatMap @ fs2.Compiler$Target.flatMap(Compiler.scala:163)
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Tue, May 23, 2023 at 5:19 PM Mike Beckerle <mbeckerle@apache.org
>> >
>> > > wrote:
>> > > >
>> > > > > I really am trying to eval the RC2.
>> > > > >
>> > > > > Right now my VMWare Linux system has been broken by recent
>> updates to
>> > > the
>> > > > > MS Windows host environment.
>> > > > >
>> > > > > I don't exactly understand this, but the VSCode front end can no
>> longer
>> > > > > connect to the server at localhost:4711. I get ECONNREFUSED. This
>> > > started
>> > > > > when a bunch of MS-Windows updates were done on the host OS. I run
>> > > VSCode
>> > > > > on the linux client machine, which lives behind a NAT firewall
>> created
>> > > by
>> > > > > VMWare workstation - so I really don't get how these could be
>> > > interacting
>> > > > > this way, but... there it is.
>> > > > >
>> > > > > I will switch back to my standalone linux PC if this isn't
>> resolved
>> > > today.
>> > > > >
>> > > > >
>> > > > > On Tue, May 16, 2023 at 1:24 PM Shane Dell <sh...@apache.org>
>> > > wrote:
>> > > > >
>> > > > >> Hello all,I'd like to call a vote to release Apache Daffodil VS
>> Code
>> > > > >> 1.3.0-rc2.
>> > > > >>
>> > > > >> All distribution packages, including signatures, digests, etc.
>> can be
>> > > > >> found at:
>> > > > >>
>> > >
>> https://dist.apache.org/repos/dist/dev/daffodil/daffodil-vscode/1.3.0-rc2
>> > > > >>
>> > > > >> This release has been signed with PGP key
>> > > > >> 86DDE7B41291E380237934F007570D3ADC76D51B, corresponding
>> > > > >> to shanedell@apache.org, which is included in the KEYS file
>> here:
>> > > > >> https://downloads.apache.org/daffodil/KEYS
>> > > > >>
>> > > > >> The release candidate has been tagged in git with 1.3.0-rc2. The
>> > > binaries
>> > > > >>
>> > > > >> For reference, here is a list of all closed GitHub issues tagged
>> with
>> > > > >> 1.3.0:
>> > > > >> https://github.com/apache/daffodil-vscode/milestone/4?closed=1
>> > > > >>
>> > > > >> Please review and vote. The vote will be open for at least 72
>> hours
>> > > > >> (Friday, 19 May 2023, 1:30pm EST).
>> > > > >>
>> > > > >> [ ] +1 approve
>> > > > >> [ ] +0 no opinion
>> > > > >> [ ] -1 disapprove (and reason why)
>> > > > >>
>> > > > >> Documentation for 1.3.0 can be found here
>> > > > >> Apache Daffodil Extension for Visual Studio Code v1.3.0 Wiki
>> > > > >> <
>> > > > >>
>> > >
>> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0
>> > > > >> >.
>> > > > >>
>> > > > >> Please note one of the large features that was focused on in
>> 1.3.0 was
>> > > > >> improving the data editor. To open the data editor you will need
>> to
>> > > open
>> > > > >> the
>> > > > >> command palette using:
>> > > > >>
>> > > > >> - Ctrl + Shift + P (windows/linux)
>> > > > >> - Command + Shift + P (mac)
>> > > > >>
>> > > > >> Then typing "data.edit", make sure "Daffodil Debug: Data Editor"
>> is
>> > > > >> selected
>> > > > >> and then hitting enter. By default the data editor will run on
>> port
>> > > 9000
>> > > > >> of
>> > > > >> your machine. However, if you want to use a different port follow
>> > > steps
>> > > > >> here
>> > > > >> Data Editor Launch Settings
>> > > > >> <
>> > > > >>
>> > >
>> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0#data-editor-launch-settings
>> > > > >> >
>> > > > >>
>> > > > >> Thank you,
>> > > > >>
>> > > > >> - Shane Dell
>> > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: GPT-4 coding assist - was: Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Posted by Davin Shearer <sc...@gmail.com>.
Hi Steve, thanks for this question as it gets to the core of whether the
solution actually matches the problem.
One detail I want to get out of the way first: The debugger and Ωedit
Scala servers require only Java 8 for the VS Code extension, you can verify
this by reviewing the CI on the project's GitHub site (see for example
https://github.com/apache/daffodil-vscode/blob/main/.github/workflows/CI.yml#L101)
that builds tests and packages the extension. The previous release used
Java 11 only because there was a debugger server dependency that wasn't
available on Java 8, but we were able to work around that in version 1.3.0
and now we ship Java 8 bytecode. Java 8 compatibility is done to maximize
the JREs that can run the software -- I think this is a reasonable
definition of all, but there will be some differences among them with
different settings and tunables.
What are the benefits of bundling a JRE:
- We get to select the precise make and version of the JRE rather than be
at the mercy of whatever is installed on the user's system
- We no longer need to make a JRE a prerequisite, so it eases the install
and maintenance burden on the user (or administrator)
- The build and test environment will now match the user's runtime
environment
- If we control the make and version of the JRE, reproducing and debugging
problems ought to be easier
Is this _required_ to solve Mike's problems? Maybe (we don't test
in Mike's environment, which is why we need to control the JRE as much as
we can), but it's not sufficient. I think the most pressing problem is to
make sure we isolate the JRE environment so it's not being influenced by
the parent VS Code environment. If we do any bundling, I'm hoping we can
do so in a later release and that.we can get 1.3.0 released with some
lighter touches.
As for https://issues.apache.org/jira/browse/LEGAL-488, I'm also not clear
on the resolution, We would be distributing the binaries as provided by
adoptium.net without modification, so I think providing a copy of
the license and a link to the source ought to suffice, but I'm _not_ a
legal expert.
I hope this helps.
On Mon, Jun 5, 2023 at 2:08 PM Steve Lawrence <sl...@apache.org> wrote:
> Temurin falls under the same license:
>
> https://adoptium.net/docs/faq/#_is_temurin_free_to_use
>
> What are the JVM environment issues that bundling solves? Mike said that
> it requires Java 11, but it's not clear what Java 11 only features the
> debugger uses. I image there is an alternative to these features that
> works on all versions of Java (for a reasonable definition of all).
>
>
>
> On 2023-06-05 01:47 PM, Davin Shearer wrote:
> > We wouldn't bundle OpenJDK, we would bundle Temurin (see
> > https://adoptium.net/). The debugger and the Ωedit Scala servers are
> both
> > compiled using Java 8. At issue is control over the JVM environment and
> > what our options are to control some of the issues that Mike faced in the
> > RC2 evaluation.
> >
> > On Mon, Jun 5, 2023 at 12:39 PM Steve Lawrence <sl...@apache.org>
> wrote:
> >
> >> Can you detail what in the VS Code extension is specific to a particular
> >> Java version, and why that can't be fixed? It feels like that would be
> >> much less work than trying to bundle and distribute a JVM.
> >>
> >> There may even be issues with ASF and distributing OpenJDK. It looks
> >> like the license is GPLv2 which ASF considers category X. I found this
> >> ticket where netbeans wanted to distributed it here:
> >>
> >> https://issues.apache.org/jira/browse/LEGAL-488
> >>
> >> It's not clear what the resolution of that bug is to me
> >>
> >> - Steve
> >>
> >>
> >> On 2023-06-05 12:21 PM, Davin Shearer wrote:
> >>> Hi Mike,
> >>>
> >>> The answer is useful. It allows the VS Code user to select the version
> >> of
> >>> Java they wish to use and it will attempt to locate a compatible JVM
> >>> through various means. Sounds really useful for Java _development_ but
> >> not
> >>> so much for other extensions to leverage for their JVM dependencies
> >> unless
> >>> they are to be used _in support of Java development_. It's also
> >> important
> >>> to note that this support is only for Java 17+ (
> >>> https://marketplace.visualstudio.com/items?itemName=redhat.java). I
> >> don't
> >>> think this is a good option for our use case.
> >>>
> >>> GraalVM looks promising though I haven't used it before, so it's hard
> for
> >>> me to predict how successful that integration would be and how long it
> >>> would take to get up and running. The idea of having native machine
> code
> >>> applications has a lot of appeal. There is even support for C/C++ code
> >>> with the LLVM compiler. The Ωedit Scala server is linking to a C
> library
> >>> using it's foreign function interface (FFI), so I'm sure how that plays
> >> out
> >>> with GraalVM. GraalVM is GPL with a ClassPath exception. We won't be
> >>> distributing GraalVM, just the executables that come out (like
> gcc/g++),
> >> so
> >>> I don't foresee licensing problems. Does anyone have any experience
> with
> >>> GraalVM?
> >>>
> >>> I think the most straight-forward path is to bundle Temurin (
> >>> https://adoptium.net/), probably using version 17 LTS. We'll be able
> to
> >>> sync the version we build on with the version we test on, distribute
> and
> >>> run on. The downside is increased package size and complexity, but
> >>> debugging issues ought to be easier since more variables are controlled
> >>> this way.
> >>>
> >>> Either of these options will take a decent chunk of research,
> >> development,
> >>> and testing. The JVM bundling option is less opaque to me, though
> >>> the GraalVM option could _potentially_ provide a superior user
> >> experience.
> >>>
> >>>
> >>> On Mon, Jun 5, 2023 at 9:53 AM Mike Beckerle <mb...@apache.org>
> >> wrote:
> >>>
> >>>> So it seems it went round full circle by telling you yes, you could
> >> bundle
> >>>> the JVM, and this redhat java language thing does so, but then they
> tell
> >>>> you it doesn't bundle the JVM anyway, it just uses the system one.
> >>>>
> >>>> So was this answer useful, or just misleading?
> >>>>
> >>>
> >>
> >>
> >
>
>
Re: GPT-4 coding assist - was: Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Posted by Steve Lawrence <sl...@apache.org>.
Temurin falls under the same license:
https://adoptium.net/docs/faq/#_is_temurin_free_to_use
What are the JVM environment issues that bundling solves? Mike said that
it requires Java 11, but it's not clear what Java 11 only features the
debugger uses. I image there is an alternative to these features that
works on all versions of Java (for a reasonable definition of all).
On 2023-06-05 01:47 PM, Davin Shearer wrote:
> We wouldn't bundle OpenJDK, we would bundle Temurin (see
> https://adoptium.net/). The debugger and the Ωedit Scala servers are both
> compiled using Java 8. At issue is control over the JVM environment and
> what our options are to control some of the issues that Mike faced in the
> RC2 evaluation.
>
> On Mon, Jun 5, 2023 at 12:39 PM Steve Lawrence <sl...@apache.org> wrote:
>
>> Can you detail what in the VS Code extension is specific to a particular
>> Java version, and why that can't be fixed? It feels like that would be
>> much less work than trying to bundle and distribute a JVM.
>>
>> There may even be issues with ASF and distributing OpenJDK. It looks
>> like the license is GPLv2 which ASF considers category X. I found this
>> ticket where netbeans wanted to distributed it here:
>>
>> https://issues.apache.org/jira/browse/LEGAL-488
>>
>> It's not clear what the resolution of that bug is to me
>>
>> - Steve
>>
>>
>> On 2023-06-05 12:21 PM, Davin Shearer wrote:
>>> Hi Mike,
>>>
>>> The answer is useful. It allows the VS Code user to select the version
>> of
>>> Java they wish to use and it will attempt to locate a compatible JVM
>>> through various means. Sounds really useful for Java _development_ but
>> not
>>> so much for other extensions to leverage for their JVM dependencies
>> unless
>>> they are to be used _in support of Java development_. It's also
>> important
>>> to note that this support is only for Java 17+ (
>>> https://marketplace.visualstudio.com/items?itemName=redhat.java). I
>> don't
>>> think this is a good option for our use case.
>>>
>>> GraalVM looks promising though I haven't used it before, so it's hard for
>>> me to predict how successful that integration would be and how long it
>>> would take to get up and running. The idea of having native machine code
>>> applications has a lot of appeal. There is even support for C/C++ code
>>> with the LLVM compiler. The Ωedit Scala server is linking to a C library
>>> using it's foreign function interface (FFI), so I'm sure how that plays
>> out
>>> with GraalVM. GraalVM is GPL with a ClassPath exception. We won't be
>>> distributing GraalVM, just the executables that come out (like gcc/g++),
>> so
>>> I don't foresee licensing problems. Does anyone have any experience with
>>> GraalVM?
>>>
>>> I think the most straight-forward path is to bundle Temurin (
>>> https://adoptium.net/), probably using version 17 LTS. We'll be able to
>>> sync the version we build on with the version we test on, distribute and
>>> run on. The downside is increased package size and complexity, but
>>> debugging issues ought to be easier since more variables are controlled
>>> this way.
>>>
>>> Either of these options will take a decent chunk of research,
>> development,
>>> and testing. The JVM bundling option is less opaque to me, though
>>> the GraalVM option could _potentially_ provide a superior user
>> experience.
>>>
>>>
>>> On Mon, Jun 5, 2023 at 9:53 AM Mike Beckerle <mb...@apache.org>
>> wrote:
>>>
>>>> So it seems it went round full circle by telling you yes, you could
>> bundle
>>>> the JVM, and this redhat java language thing does so, but then they tell
>>>> you it doesn't bundle the JVM anyway, it just uses the system one.
>>>>
>>>> So was this answer useful, or just misleading?
>>>>
>>>
>>
>>
>
Re: GPT-4 coding assist - was: Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Posted by Davin Shearer <sc...@gmail.com>.
We wouldn't bundle OpenJDK, we would bundle Temurin (see
https://adoptium.net/). The debugger and the Ωedit Scala servers are both
compiled using Java 8. At issue is control over the JVM environment and
what our options are to control some of the issues that Mike faced in the
RC2 evaluation.
On Mon, Jun 5, 2023 at 12:39 PM Steve Lawrence <sl...@apache.org> wrote:
> Can you detail what in the VS Code extension is specific to a particular
> Java version, and why that can't be fixed? It feels like that would be
> much less work than trying to bundle and distribute a JVM.
>
> There may even be issues with ASF and distributing OpenJDK. It looks
> like the license is GPLv2 which ASF considers category X. I found this
> ticket where netbeans wanted to distributed it here:
>
> https://issues.apache.org/jira/browse/LEGAL-488
>
> It's not clear what the resolution of that bug is to me
>
> - Steve
>
>
> On 2023-06-05 12:21 PM, Davin Shearer wrote:
> > Hi Mike,
> >
> > The answer is useful. It allows the VS Code user to select the version
> of
> > Java they wish to use and it will attempt to locate a compatible JVM
> > through various means. Sounds really useful for Java _development_ but
> not
> > so much for other extensions to leverage for their JVM dependencies
> unless
> > they are to be used _in support of Java development_. It's also
> important
> > to note that this support is only for Java 17+ (
> > https://marketplace.visualstudio.com/items?itemName=redhat.java). I
> don't
> > think this is a good option for our use case.
> >
> > GraalVM looks promising though I haven't used it before, so it's hard for
> > me to predict how successful that integration would be and how long it
> > would take to get up and running. The idea of having native machine code
> > applications has a lot of appeal. There is even support for C/C++ code
> > with the LLVM compiler. The Ωedit Scala server is linking to a C library
> > using it's foreign function interface (FFI), so I'm sure how that plays
> out
> > with GraalVM. GraalVM is GPL with a ClassPath exception. We won't be
> > distributing GraalVM, just the executables that come out (like gcc/g++),
> so
> > I don't foresee licensing problems. Does anyone have any experience with
> > GraalVM?
> >
> > I think the most straight-forward path is to bundle Temurin (
> > https://adoptium.net/), probably using version 17 LTS. We'll be able to
> > sync the version we build on with the version we test on, distribute and
> > run on. The downside is increased package size and complexity, but
> > debugging issues ought to be easier since more variables are controlled
> > this way.
> >
> > Either of these options will take a decent chunk of research,
> development,
> > and testing. The JVM bundling option is less opaque to me, though
> > the GraalVM option could _potentially_ provide a superior user
> experience.
> >
> >
> > On Mon, Jun 5, 2023 at 9:53 AM Mike Beckerle <mb...@apache.org>
> wrote:
> >
> >> So it seems it went round full circle by telling you yes, you could
> bundle
> >> the JVM, and this redhat java language thing does so, but then they tell
> >> you it doesn't bundle the JVM anyway, it just uses the system one.
> >>
> >> So was this answer useful, or just misleading?
> >>
> >
>
>
Re: GPT-4 coding assist - was: Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Posted by Steve Lawrence <sl...@apache.org>.
Can you detail what in the VS Code extension is specific to a particular
Java version, and why that can't be fixed? It feels like that would be
much less work than trying to bundle and distribute a JVM.
There may even be issues with ASF and distributing OpenJDK. It looks
like the license is GPLv2 which ASF considers category X. I found this
ticket where netbeans wanted to distributed it here:
https://issues.apache.org/jira/browse/LEGAL-488
It's not clear what the resolution of that bug is to me
- Steve
On 2023-06-05 12:21 PM, Davin Shearer wrote:
> Hi Mike,
>
> The answer is useful. It allows the VS Code user to select the version of
> Java they wish to use and it will attempt to locate a compatible JVM
> through various means. Sounds really useful for Java _development_ but not
> so much for other extensions to leverage for their JVM dependencies unless
> they are to be used _in support of Java development_. It's also important
> to note that this support is only for Java 17+ (
> https://marketplace.visualstudio.com/items?itemName=redhat.java). I don't
> think this is a good option for our use case.
>
> GraalVM looks promising though I haven't used it before, so it's hard for
> me to predict how successful that integration would be and how long it
> would take to get up and running. The idea of having native machine code
> applications has a lot of appeal. There is even support for C/C++ code
> with the LLVM compiler. The Ωedit Scala server is linking to a C library
> using it's foreign function interface (FFI), so I'm sure how that plays out
> with GraalVM. GraalVM is GPL with a ClassPath exception. We won't be
> distributing GraalVM, just the executables that come out (like gcc/g++), so
> I don't foresee licensing problems. Does anyone have any experience with
> GraalVM?
>
> I think the most straight-forward path is to bundle Temurin (
> https://adoptium.net/), probably using version 17 LTS. We'll be able to
> sync the version we build on with the version we test on, distribute and
> run on. The downside is increased package size and complexity, but
> debugging issues ought to be easier since more variables are controlled
> this way.
>
> Either of these options will take a decent chunk of research, development,
> and testing. The JVM bundling option is less opaque to me, though
> the GraalVM option could _potentially_ provide a superior user experience.
>
>
> On Mon, Jun 5, 2023 at 9:53 AM Mike Beckerle <mb...@apache.org> wrote:
>
>> So it seems it went round full circle by telling you yes, you could bundle
>> the JVM, and this redhat java language thing does so, but then they tell
>> you it doesn't bundle the JVM anyway, it just uses the system one.
>>
>> So was this answer useful, or just misleading?
>>
>
Re: GPT-4 coding assist - was: Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Posted by Davin Shearer <sc...@gmail.com>.
Hi Mike,
The answer is useful. It allows the VS Code user to select the version of
Java they wish to use and it will attempt to locate a compatible JVM
through various means. Sounds really useful for Java _development_ but not
so much for other extensions to leverage for their JVM dependencies unless
they are to be used _in support of Java development_. It's also important
to note that this support is only for Java 17+ (
https://marketplace.visualstudio.com/items?itemName=redhat.java). I don't
think this is a good option for our use case.
GraalVM looks promising though I haven't used it before, so it's hard for
me to predict how successful that integration would be and how long it
would take to get up and running. The idea of having native machine code
applications has a lot of appeal. There is even support for C/C++ code
with the LLVM compiler. The Ωedit Scala server is linking to a C library
using it's foreign function interface (FFI), so I'm sure how that plays out
with GraalVM. GraalVM is GPL with a ClassPath exception. We won't be
distributing GraalVM, just the executables that come out (like gcc/g++), so
I don't foresee licensing problems. Does anyone have any experience with
GraalVM?
I think the most straight-forward path is to bundle Temurin (
https://adoptium.net/), probably using version 17 LTS. We'll be able to
sync the version we build on with the version we test on, distribute and
run on. The downside is increased package size and complexity, but
debugging issues ought to be easier since more variables are controlled
this way.
Either of these options will take a decent chunk of research, development,
and testing. The JVM bundling option is less opaque to me, though
the GraalVM option could _potentially_ provide a superior user experience.
On Mon, Jun 5, 2023 at 9:53 AM Mike Beckerle <mb...@apache.org> wrote:
> So it seems it went round full circle by telling you yes, you could bundle
> the JVM, and this redhat java language thing does so, but then they tell
> you it doesn't bundle the JVM anyway, it just uses the system one.
>
> So was this answer useful, or just misleading?
>
GPT-4 coding assist - was: Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Posted by Mike Beckerle <mb...@apache.org>.
So it seems it went round full circle by telling you yes, you could bundle
the JVM, and this redhat java language thing does so, but then they tell
you it doesn't bundle the JVM anyway, it just uses the system one.
So was this answer useful, or just misleading?
Re: [VOTE] Apache Daffodil VS Code Extension 1.3.0-rc2
Posted by Mike Beckerle <mb...@apache.org>.
I'll create the tickets for the individual issues in my review.
On Fri, Jun 2, 2023 at 3:24 PM Davin Shearer <sc...@gmail.com> wrote:
> Thanks for the detailed update and your patience Mike. This is
> definitely not the user experience we're hoping for. Are these regressions
> that have occurred since the v1.2.0 release?
>
> I'm a big fan of compartmentalization and technologies like docker and
> python's venv (virtual environment) among many others. Relying on system
> installs of virtual machines (e.g. Java's JVM) and interpreters (e.g.,
> Python, Perl, etc) introduces uncontrolled variables into our runtime
> environment, which I believe is one of the major encumbrances that you're
> experiencing and something I hope we can address as a community.
>
> I don't think docker is the right answer here because that introduces
> another system-level dependency... the docker service. I think packaging
> a JVM or pulling it in as a VS Code dependency (not a system level
> dependency like docker) will give us much tighter control over our runtime
> environment. We'll need to be careful too about how we execute our
> services on that JVM so its execution environment is properly
> compartmentalized. In addition to the Daffodil debugger service, the Data
> Editor part of the extension also uses a service running on the JVM which
> will benefit from this compartmentalization. This also will allow us to
> upgrade from Java 8 to something more recent (this will help with domain
> sockets and named fifos too). Anyway, that's the direction I'd be inclined
> to head in, but it's definitely worth a discussion and hopefully we can
> converge on solid design. I had an interesting conversation on this topic
> with GPT-4 that I think is worth sharing (see:
> https://chat.openai.com/share/da2ca068-714a-4eec-8a0e-8117069b3125).
>
> Each distinct failure that you've encountered in testing RC2 needs to have
> issues associated with them on the project's GitHub so we can track, triage
> and prioritize, schedule, and assign them. If you're willing to enter your
> issues into the GitHub tracker, that would be greatly appreciated and
> you'll be able to stay in the loop on how those issues progress and help us
> in the review phase to make sure that each issue is being properly and
> fully addressed.
>
> Shane is out, but I'm sure he'll have some additional follow up and things
> to try before making new pull requests.
>
>
> On Wed, May 31, 2023 at 7:02 PM Mike Beckerle <mb...@apache.org>
> wrote:
>
> >
> > So, I have the 1.3.0 RC2 vscode extension debugger working now for PCAP
> w/
> > ethernetIP and I can set a breakpoint in ethernetIP.dfdl.xsd and I no
> > longer get an NPE.
> >
> > Configuring this is really far too difficult, and I have to have the
> > schema working first with sbt in order to even hope to configure it.
> >
> > But I'll get back to that later.
> >
> > More basic issue is that I am unable to debug even dead-simple errors
> with
> > it. I clobbered the magic number in the first bytes of an icmp1.cap file.
> > This fails an assert at line 97 of pcap.dfdl.xsd, which is the first
> > element of simple type.
> >
> > I step through with the debugger, and nothing displays *anything* about
> > the assert failure. Execution ends. A little pop-up dialog says it wrote
> my
> > infoset out, which it didn't. There's nothing showing that anything went
> > wrong. All panels on the left are blank.
> >
> > If I modify the schema and make any mistake, then I get a blocking dialog
> > showing the SDEs. This is basically intolerable, and is why I have to go
> > back to sbt to debug this. You can't even go through the SDEs as you move
> > the editor. It's a blocking dialog.
> >
> > So both for schema compilation, and runtime errors there simply has to be
> > a non-modal display of errors, probably on the left with variables,
> watch,
> > call-stack, and breakpoints. But a quasi-edit-buffer containing them
> would
> > also be fine, or even just output them to the terminal window.
> >
> > The rest of this email is about the hoops I had to jump through to get
> > this to work at all. Arguably, these issues are about working with a
> > multi-component schema, and we could define our objectives to simpler
> > schemas than this, but that puts it in the realm of toy-only schemas. The
> > example is just PCAP + ethernetIP, which is not an extreme example.
> >
> > The existing shell environment of the terminal definitely causes
> problems.
> > But just exiting that so that the debugger starts a new terminal doesn't
> > fix it, as that inherits the environment from where vscode was started.
> It
> > starts a new login shell unless you take steps to prevent that.
> >
> > A big trip-up in usability is that the wizard that creates the
> > launch.json, doesn't actually let you populate the daffodilDebugClaspath.
> > You have to edit that by hand, and nothing verifies it, so any typo and
> you
> > get tedious failures that are hard to figure out. You just get an SDE -
> if
> > you are lucky - via a blocking dialog.
> >
> > The wizard should help you put directories and corresponding jars on the
> > daffodilDebugClasspath, and it should not let you put a non-existing
> > directory or jar file on it. Ideally, at launch time the
> > daffodilDebugClasspath should be checked, just in case something on it no
> > longer exists.
> >
> > If the debugger happens to find a file on the daffodilDebugClasspath in a
> > jar, vs. in a regular directory (maybe you have a typo in the regular
> > directory path), so it passes that and hits the jar, then setting a
> > breakpoint causes a NPE. Something requires a "hierarchical URL" which a
> > jar-file URL is not. This is the bug that really has to be fixed so that
> > you can set breakpoints regardless of whether the source DFDL schema file
> > was in a jar or not.
> >
> > Note: at Owl we have DFDL schemas where the starting root element of the
> > schema isn't even part of the schema, it's part of a component schema
> that
> > is in a jar on the classpath. This happens when you have an
> > envelope-payload idiom. The schema that combines the envelope and payload
> > together contains only the glue part that connects the two together. The
> > root element is the envelope's root element, and that schema is in a jar
> > file.
> >
> > I did not see any feature in launch.json for specifying the
> name/namespace
> > of the root element. There is just the "program" element, which must be
> to
> > a file in the current schema. That needs to be able to be specified just
> as
> > element name + namespace.
> >
> > The jars have to be on the daffodilDebugClasspath because of the plugins
> > that daffodil must find in them. That's understandable, the problem is
> > having to also have a regular file location for any schema you want to be
> > able to put a breakpoint in and it must be before the corresponding jar.
> >
> > That all said, here are the scripts and settings I used to get it to
> work:
> >
> > First, I have this script I use to launch vscode in a minimal/empty
> > environment. This technique is useful to
> > determine if you are depending on the environment in any way:
> >
> > #! /bin/bash
> > if [ "$1" = "emptyEnv" ] # is it the special first arg
> > then
> > shift # discard special first arg, and continue with the rest of the
> > script
> > else
> > CMD=$0
> > exec -c $CMD "emptyEnv" $* # exec with an empty environment, special
> > first arg
> > fi
> > # the environment is empty when we get here
> > # just add env vars we need to get vscode to work
> > export LANG=en_US.UTF-8
> > export HOME=/home/mbeckerle
> > export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64 # java 11 required
> > export PATH=$JAVA_HOME/bin:$PATH
> > export DISPLAY=:0
> > export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus # vscode
> > spits lots of errors without this
> > exec /usr/bin/code
> > # end
> >
> > Then, within ~/.config/Code/User/settings.json, I had to create these
> > settings for terminal:
> >
> > "terminal.integrated.inheritEnv": false,
> > "terminal.integrated.profiles.linux": {
> > "Custom Init": {
> > "path": "/bin/bash",
> > "args": [
> > "--norc"
> > ]
> > }
> > },
> > "terminal.integrated.defaultProfile.linux": "Custom Init"
> >
> > That instructs vscode not to create a login shell for the terminal. The
> > --norc option means it does not run the .bashrc file so that the java
> > version specified in the launch script sticks.
> >
> > I am not sure the inheritEnv setting actually does anything. That alone
> > sounds like the right thing, but that alone didn't solve the problem.
> > You still have everything in the login shell environment where vscode was
> > launched. Hence the need for the environment-emptying launch script.
> >
> > With those two things, a terminal created by vscode does not inherit
> > everything my typical shell has. For example there is no daffodil on the
> > path, no DAFFODIL_CLASSPATH env var, etc.
> >
> > Given the above if I just create a terminal with Terminal > new Terminal:
> > The environment is relatively simple:
> >
> > bash-5.0$ env
> > COLORTERM=truecolor
> > TERM_PROGRAM_VERSION=1.78.2
> > JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64
> > NO_AT_BRIDGE=1
> > PWD=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP
> > VSCODE_GIT_ASKPASS_NODE=/usr/share/code/code
> > HOME=/home/mbeckerle
> > LANG=en_US.UTF-8
> > GIT_ASKPASS=/usr/share/code/resources/app/extensions/git/dist/askpass.sh
> > CHROME_DESKTOP=code-url-handler.desktop
> > VSCODE_GIT_ASKPASS_EXTRA_ARGS=--ms-enable-electron-run-as-node
> > TERM=xterm-256color
> > VSCODE_GIT_IPC_HANDLE=/tmp/vscode-git-56a269c0f5.sock
> > DISPLAY=:0
> > SHLVL=1
> >
> >
> VSCODE_GIT_ASKPASS_MAIN=/usr/share/code/resources/app/extensions/git/dist/askpass-main.js
> > GDK_BACKEND=x11
> >
> >
> PATH=/usr/lib/jvm/java-11-openjdk-amd64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
> > ORIGINAL_XDG_CURRENT_DESKTOP=undefined
> > DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
> > TERM_PROGRAM=vscode
> > _=/usr/bin/env
> >
> > If I run daffodil, then c-C in the terminal it creates, then the
> > environment has this added environment variable only.
> >
> >
> >
> DAFFODIL_DEBUG_CLASSPATH=/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/../ethernetIP/src/main/resources:/home/mbeckerle/dataiti/opensource/DFDLSchemas/PCAP/lib_managed/jars/com.owlcyberdefense/dfdl-ethernetip/dfdl-ethernetip-1.2.0.jar
> >
> > This works. My launch.json is attached.
> >
> >
> >
> >
> > On Fri, May 26, 2023 at 2:06 PM Shane Dell <sh...@apache.org> wrote:
> >
> >> Mike,
> >>
> >> The VSCode Extension should be using the daffodil command that is
> bundled
> >> with the
> >> extension. The issue might still be with that shell env variable you
> have
> >> for
> >> DAFFODIL_CLASSPATH, but this might only be an issue with a terminal that
> >> is already opened
> >> with the name "daffodil-debugger". If you close that terminal, the next
> >> you time you run it should
> >> create a new terminal with the env set properly. If you could try this
> >> and see if it works how you
> >> expect, that would help us single out this issue out to an already
> active
> >> terminal.
> >>
> >> Thank you,
> >>
> >> Shane Dell
> >>
> >>
> >> On 2023/05/25 23:59:53 Mike Beckerle wrote:
> >> > There are so many environment interactions....
> >> >
> >> > So I have discovered that the VSCode extension just seems to run
> >> whatever
> >> > daffodil command is on the path. It's not isolated at all from the
> shell
> >> > environment.
> >> >
> >> > I found this out because I have a customized daffodil shell script
> that
> >> > setups a bunch of DAFFODIL_CLASSPATH contents, and I couldn't figure
> out
> >> > why those were part of the VSCode environment.
> >> >
> >> > Shouldn't vscode run exactly the daffodil bundled with it, unless you
> >> > explicitly configure it not to?
> >> >
> >> > That's why it is finding the ethernetIP.dfdl.xsd file in the jar, not
> in
> >> > the file system. Because my daffodil command sets up lots of jars on
> the
> >> > DAFFODIL_CLASSPATH.
> >> >
> >> > It also depends on Java 11, breaks with Java 17. I also don't think
> this
> >> > should be inherited from the environment.
> >> >
> >> > I really have a hard time using, or even testing the VSCode extension
> >> if I
> >> > have to scrub my shell environment of everything in it in order to
> make
> >> the
> >> > vscode extension work.
> >> >
> >> > But it is this kind of problem that caused people to invent docker
> >> > containers.
> >> >
> >> > I am going to make a launcher script for VSCode that empties the
> >> > environment, then supplies only what is absolutely necessary, with no
> >> > outside environment assumptions.
> >> >
> >> > I will try to do this Friday if possible.
> >> >
> >> >
> >> >
> >> > On Wed, May 24, 2023 at 2:50 PM Shane Dell <sh...@apache.org>
> >> wrote:
> >> >
> >> > > Mike,
> >> > >
> >> > > I am not sure why you would be seeing this. I am able to set the
> >> > > breakpoint where you
> >> > > mentioned without any issues. Can you try removing the daffodil-dap
> >> > > folder? This should be
> >> > > somewhere like:
> >> > > /home/$USER/.local/share/daffodil-dap
> >> > > The issue I am thinking is possibly a bug with the rc1 debugger that
> >> may
> >> > > not be present on rc2.
> >> > > But if the folder daffodil-dap/daffodil-debugger-3.4.0-1.3.0/
> already
> >> > > exists from rc1 then rc2 will
> >> > > not remake it. So in order to remake it you have to manually remove
> >> this
> >> > > folder. Let me know if
> >> > > this helps.
> >> > >
> >> > > Thank you,
> >> > >
> >> > > Shane Dell
> >> > >
> >> > > On 2023/05/24 15:15:33 Mike Beckerle wrote:
> >> > > > I have been able to continue evaluating RC2.
> >> > > >
> >> > > > My vote is -1
> >> > > >
> >> > > > My testing steps listed below. I am trying to debug PCAP. I have a
> >> broken
> >> > > > icmp packet (bad magic number), and also just step through a
> >> correct icmp
> >> > > > packet.
> >> > > >
> >> > > > I am trying to debug icmp.badMagicNum.cap data file.
> >> > > >
> >> > > > I should get a failure of an assert because the magic number is
> not
> >> one
> >> > > of
> >> > > > the expected values.
> >> > > >
> >> > > > I'm just trying to single-step through it until it hits the error.
> >> > > >
> >> > > > Once we get to the MagicNumber element, it doesn't position on the
> >> assert
> >> > > > while evaluating that expression or even on the element carrying
> the
> >> > > assert.
> >> > > > It jumps back to the top-level xs:schema position.
> >> > > >
> >> > > > I continue to click single step. Watching for the error to be
> >> displayed
> >> > > in
> >> > > > the upper left under Variables > Parse > Diagnostics > Errors
> which
> >> is
> >> > > > where I think it ought to go.
> >> > > >
> >> > > > At some point the failure is somehow detected as the debug session
> >> ends.
> >> > > A
> >> > > > dialog offers to open /tmp/infoset.xml for me, which I open and it
> >> is
> >> > > empty
> >> > > > of course.
> >> > > >
> >> > > > Nothing displayed the error message from the assert, or even
> >> indicated
> >> > > what
> >> > > > went wrong at all. Not the state display in the upper left, not
> the
> >> > > > terminal, nor debug console nor output.
> >> > > >
> >> > > > The upper left box with "Variables" is cleared at the end. It
> should
> >> > > > certainly not be cleared at the end of a run when final state may
> >> be of
> >> > > > interest. At the start of a run, or manually, but not at the end
> of
> >> a
> >> > > run.
> >> > > >
> >> > > > Second attempt to use it:
> >> > > >
> >> > > > I create a second launch config this time for what should be a
> >> successful
> >> > > > parse of icmp1.cap
> >> > > >
> >> > > > As I single step, the position of the cursor on the schema is
> >> jumping
> >> > > > around all over the place back and forth between a containing
> >> element and
> >> > > > the child elements it contains.
> >> > > > The infoset display does not get updated until long after the
> >> display has
> >> > > > moved on and is positioned on the next element.
> >> > > >
> >> > > > This jumping around gets completely out of hand once you step into
> >> the
> >> > > > complexType "Ethernet" which is in the ethernetIP jar file.
> >> > > >
> >> > > > At that point, after each child of Ethernet, it jumps back to the
> >> > > Ethernet
> >> > > > element in the pcap.dfdl.xsd file, and then back into the
> >> > > > ethernetIP.dfdl.xsd for the next child, then back to the
> >> pcap.dfdl.xsd
> >> > > > file, etc.
> >> > > >
> >> > > > All this jumping around basically makes stepping useless. So I
> >> decide to
> >> > > > put a breakpoint on the Version element inside the IPv4Header
> >> element
> >> > > > inside ethernetIP.dfdl.xsd.
> >> > > >
> >> > > > Just adding the breakpoint causes the debug session to end with an
> >> NPE in
> >> > > > the dap setBreakpoint.
> >> > > >
> >> > > > ERROR o.a.d.d.d.DAPodil - unhandled error
> >> > > > java.lang.NullPointerException: null
> >> > > > at
> >> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)
> >> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
> >> > > > at
> >> > > >
> java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)
> >> > > > at java.base/java.nio.file.Path.of(Path.java:147)
> >> > > > at java.base/java.nio.file.Paths.get(Paths.java:69)
> >> > > > at
> >> > > >
> >> > >
> >>
> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> >> > > >
> >> > >
> >>
> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at handleErrorWith @
> >> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> >> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > 2023-05-24 11:09:10,717 [io-compute-1] DEBUG o.a.d.d.d.DAPodil -
> >> > > whenDone:
> >> > > > completed
> >> > > > java.lang.NullPointerException
> >> > > > at
> >> > > java.base/sun.nio.fs.UnixPath.normalizeAndCheck(UnixPath.java:75)
> >> > > > at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
> >> > > > at
> >> > > >
> java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:279)
> >> > > > at java.base/java.nio.file.Path.of(Path.java:147)
> >> > > > at java.base/java.nio.file.Paths.get(Paths.java:69)
> >> > > > at
> >> > > >
> >> > >
> >>
> org.apache.daffodil.debugger.dap.DAPodil.$anonfun$setBreakpoints$1(DAPodil.scala:267)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> >> > > >
> >> > >
> >>
> org.apache.daffodil.debugger.dap.DAPodil.setBreakpoints(DAPodil.scala:263)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at handleErrorWith @
> >> > > > fs2.Compiler$Target.handleErrorWith(Compiler.scala:161)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @ fs2.Pull$.goEval$1(Pull.scala:1057)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > > at flatMap @
> >> fs2.Pull$.fs2$Pull$$interruptGuard$1(Pull.scala:929)
> >> > > > at get @ fs2.internal.Scope.openScope(Scope.scala:281)
> >> > > > at flatMap @
> fs2.Compiler$Target.flatMap(Compiler.scala:163)
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Tue, May 23, 2023 at 5:19 PM Mike Beckerle <
> mbeckerle@apache.org
> >> >
> >> > > wrote:
> >> > > >
> >> > > > > I really am trying to eval the RC2.
> >> > > > >
> >> > > > > Right now my VMWare Linux system has been broken by recent
> >> updates to
> >> > > the
> >> > > > > MS Windows host environment.
> >> > > > >
> >> > > > > I don't exactly understand this, but the VSCode front end can no
> >> longer
> >> > > > > connect to the server at localhost:4711. I get ECONNREFUSED.
> This
> >> > > started
> >> > > > > when a bunch of MS-Windows updates were done on the host OS. I
> run
> >> > > VSCode
> >> > > > > on the linux client machine, which lives behind a NAT firewall
> >> created
> >> > > by
> >> > > > > VMWare workstation - so I really don't get how these could be
> >> > > interacting
> >> > > > > this way, but... there it is.
> >> > > > >
> >> > > > > I will switch back to my standalone linux PC if this isn't
> >> resolved
> >> > > today.
> >> > > > >
> >> > > > >
> >> > > > > On Tue, May 16, 2023 at 1:24 PM Shane Dell <
> shanedell@apache.org>
> >> > > wrote:
> >> > > > >
> >> > > > >> Hello all,I'd like to call a vote to release Apache Daffodil VS
> >> Code
> >> > > > >> 1.3.0-rc2.
> >> > > > >>
> >> > > > >> All distribution packages, including signatures, digests, etc.
> >> can be
> >> > > > >> found at:
> >> > > > >>
> >> > >
> >>
> https://dist.apache.org/repos/dist/dev/daffodil/daffodil-vscode/1.3.0-rc2
> >> > > > >>
> >> > > > >> This release has been signed with PGP key
> >> > > > >> 86DDE7B41291E380237934F007570D3ADC76D51B, corresponding
> >> > > > >> to shanedell@apache.org, which is included in the KEYS file
> >> here:
> >> > > > >> https://downloads.apache.org/daffodil/KEYS
> >> > > > >>
> >> > > > >> The release candidate has been tagged in git with 1.3.0-rc2.
> The
> >> > > binaries
> >> > > > >>
> >> > > > >> For reference, here is a list of all closed GitHub issues
> tagged
> >> with
> >> > > > >> 1.3.0:
> >> > > > >> https://github.com/apache/daffodil-vscode/milestone/4?closed=1
> >> > > > >>
> >> > > > >> Please review and vote. The vote will be open for at least 72
> >> hours
> >> > > > >> (Friday, 19 May 2023, 1:30pm EST).
> >> > > > >>
> >> > > > >> [ ] +1 approve
> >> > > > >> [ ] +0 no opinion
> >> > > > >> [ ] -1 disapprove (and reason why)
> >> > > > >>
> >> > > > >> Documentation for 1.3.0 can be found here
> >> > > > >> Apache Daffodil Extension for Visual Studio Code v1.3.0 Wiki
> >> > > > >> <
> >> > > > >>
> >> > >
> >>
> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0
> >> > > > >> >.
> >> > > > >>
> >> > > > >> Please note one of the large features that was focused on in
> >> 1.3.0 was
> >> > > > >> improving the data editor. To open the data editor you will
> need
> >> to
> >> > > open
> >> > > > >> the
> >> > > > >> command palette using:
> >> > > > >>
> >> > > > >> - Ctrl + Shift + P (windows/linux)
> >> > > > >> - Command + Shift + P (mac)
> >> > > > >>
> >> > > > >> Then typing "data.edit", make sure "Daffodil Debug: Data
> Editor"
> >> is
> >> > > > >> selected
> >> > > > >> and then hitting enter. By default the data editor will run on
> >> port
> >> > > 9000
> >> > > > >> of
> >> > > > >> your machine. However, if you want to use a different port
> follow
> >> > > steps
> >> > > > >> here
> >> > > > >> Data Editor Launch Settings
> >> > > > >> <
> >> > > > >>
> >> > >
> >>
> https://github.com/apache/daffodil-vscode/wiki/Apache-Daffodil%E2%84%A2-Extension-for-Visual-Studio-Code:-v1.3.0#data-editor-launch-settings
> >> > > > >> >
> >> > > > >>
> >> > > > >> Thank you,
> >> > > > >>
> >> > > > >> - Shane Dell
> >> > > > >>
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>