You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@lucenenet.apache.org by Laimonas Simutis <la...@gmail.com> on 2019/01/25 13:14:25 UTC

Update on dev happenings

Hello there,

A quick note on what was discovered/done this past week while getting
things back up and running.

- Simon's fix for https://issues.apache.org/jira/browse/LUCENENET-597 was
merged
- lucenenet-site repo is up where we will commit the website code
- old/irrelevant JIRA tickets cleaned up (more cleanup coming)

Next:

- working on getting the TeamCity builds going so that we could verify with
more confidence another of Simon's fixes (
https://github.com/apache/lucenenet/pull/218), and in general get the
feedback from builds flowing to have more confidence on the 4.8 stability.
- work with Shannon to get his PR for the website merged into the site
repo, hopefully, we hear from him soon.
- clean up CONTRIBUTE.md so it's very clear what sort of next tasks are
outstanding

Also, if you have time, take a look at the comment Shad made in this ticket
(comment highlighted):
https://issues.apache.org/jira/browse/LUCENENET-601?focusedCommentId=16465234.
I think that's a very good overview of where we are at with 4.8. Short
version: if we decide to go without fully ICU working bits, we could
potentially release 4.8. But that's a discussion for a separate thread.


Have a great Friday everyone,

Laimonas

RE: Update on dev happenings

Posted by Shad Storhaug <sh...@shadstorhaug.com>.

Patric,

Thanks for the suggestion. I am not sure how relevant it is though because our GitHub presence is just a mirror of the repo that is hosted at Apache. When we push commits, we push them directly to Apache so GitHub really has no way to hook into the process unless it is somehow based on watching for commit changes by polling.

In fact, that is why Azure DevOps doesn't "just work" - they aren't watching for commit changes to the mirror, they are expecting the commits through the front door that is strictly read-only by Apache. I haven't tried, but I suspect that the PR functionality of Azure DevOps will work because those are submitted directly through GitHub. It would be a bit of a pain for every committer to have to open a PR first before merging their own commits in order to get any CI results, though.

I don't even have enough access to the apache/lucenenet GitHub account to merge a pull request and I am not sure whether I can get it because I haven't asked. I have just been working around it by merging PRs with Git and closing them via commit message.

It seems that the path of least resistance here is to use our existing setup on TeamCity to fire an API call to Azure DevOps. TeamCity has its own built in key vault so whatever credentials are required to call the Azure DevOps API can be stored there, and whatever task that is necessary can be setup to make the API call. Its quite slow, though - the poll only happens every 20 minutes and then there could be another 5-10 minute wait to get a build agent. The polling interval can be adjusted, but not sure what the minimum would be that Apache's server could handle. Slow is still better than never or manual, though.

Thanks,
Shad Storhaug (NightOwl888)

-----Original Message-----
From: Patric Forsgard <pa...@tasteful.se> 
Sent: Friday, August 9, 2019 3:26 AM
To: dev@lucenenet.apache.org
Subject: Re: Update on dev happenings

I was notified today about Github Actions (
https://developer.github.com/actions/) that also can be executed on developer computer with ACT ( https://github.com/nektos/act/blob/master/README.md). Can this be someting to use instead? Yes, it will be a dependency on docker in that case to be able to run everyting.

// Patric

On Thu, Aug 8, 2019, 06:37 Shad Storhaug <sh...@shadstorhaug.com> wrote:

> Update.
>
> I really didn't want to give up TeamCity because it is far better than 
> Azure DevOps, but it seems that between the fact that JetBrains won't 
> give us any more than one build agent, the fact it doesn't seem to be 
> possible to get any Agents on Linux or Mac (at least none that will 
> run powershell), and the fact that Microsoft broke the ability to get 
> test results into TeamCity (I tried half a dozen configurations, but 
> due to the other issues it didn't seem worth the effort) that Azure 
> DevOps is a better choice. We definitely should switch to using Azure 
> Artifacts regardless because of the generous amount of feed storage 
> they allow (which frankly was why we didn't target .NET Standard 2.0 on all of the projects on the last release).
>
> We now have a YAML Azure DevOps pipeline in the repository. The test 
> projects for Lucene.Net (core) were split into smaller pieces in order 
> to run the tests in parallel in background jobs, and I was able to get 
> the testing time down to about 40 minutes, which is a pretty safe 
> margin less than the 1 hour limit Azure DevOps imposes on how long a 
> job can run. I set it up so basically anyone can create a free Azure 
> DevOps account, import the template, and it will automatically build, 
> version a CI build based on a counter, and test all target frameworks 
> on Windows, macOS and Ubuntu. Of course, it will build much faster if 
> you setup the project as a public project so you can get 10 parallel 
> jobs. If you enter an Azure Artifact feed ID (a GUID) as a variable, 
> it will also push the packages and debug symbols into the feed (see 
> the documentation in azure-pipelines.yml https://github.com/apache/lucenenet/blob/master/azure-pipelines.yml#L22).
>
> However, I have run into a couple of obstacles with actually getting 
> this to work with the official repo. It seems that there are INFRA 
> issues blocking it from working as a CI integration because Microsoft 
> requires write repo access and Apache has a strict policy against it:
> https://issues.apache.org/jira/browse/INFRA-17030
>
> Furthermore, Microsoft doesn't support YAML in any "Other Git (generic)"
> repositories:
> https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/?view=az
> ure-devops
> .
>
> Despite this, I was able to use the GitHub mirror of our repo by first 
> setting it up on a fork, and then switching the GitHub repo name to 
> apache/lucenenet, running in read-only mode. One problem solved.
>
> I have setup what I hope will be the official Lucene.NET organization
> here:
> https://dev.azure.com/lucene-net/Lucene.NET/_build/results?buildId=1&v
> iew=ms.vss-test-web.build-test-results-tab,
> but if we need to change it for any reason, nearly all of the 
> configuration is in YAML so it isn't a lot of work.
>
> This configuration allows us to build a release because that is 
> launched manually, but CI builds are another story. Azure DevOps 
> doesn't get triggered by a push to the Apache repo, and they don't 
> seem to have an option to poll for changes like we do in TeamCity. A 
> couple of options I have been contemplating:
>
> 1. Setup a build in TeamCity to do an API call to Azure DevOps to 
> launch the build:
> https://docs.microsoft.com/en-us/rest/api/azure/devops/build/?view=azu
> re-devops-rest-5.0#queue-a-build 2. Setup an email address and 
> register it with commits@lucenenet.apache.org and trigger the build 
> whenever an email is received
>
> However, before I hack together a solution, I wanted to check whether 
> any of the PMC know if it is possible to have INFRA setup a commit 
> hook so we can do this in a more straightforward way? Possibly by 
> setting up an Azure Function and having Apache call it via a URL:
> https://github.com/MicrosoftDocs/vsts-docs/issues/2366? I would ask 
> them, but the docs say to ask here first...
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
> -----Original Message-----
> From: Laimonas Simutis <la...@gmail.com>
> Sent: Tuesday, March 12, 2019 8:12 PM
> To: dev@lucenenet.apache.org
> Subject: Re: Update on dev happenings
>
> Quick update. I've had a successful build with azure pipelines, you 
> can see it here:
> https://dev.azure.com/lsimutis/lsimutis/_build/results?buildId=5
>
> This is off of the fork of the current repo. Will see how the tests do 
> there next and then perhaps we can seriously eval moving our builds 
> off of teamcity on jetbrains. Or at the very least get the PRs 
> building and verified automatically while we still decide if 
> resurrecting teamcity setup is worth persuing.
>
>
> Laimonas
>
> On Tue, Feb 26, 2019 at 8:35 AM Laimonas Simutis <la...@gmail.com> wrote:
>
> > I wonder if the assumption that .net core needs to be installed is 
> > outdated and we can find an agent with the .net core present.
> > psake abstraction here is hard to follow and reason what's going on...
> > I will check if there are other .net core based projects that are 
> > being built in there that we can try to follow.
> >
> > Another option is for us to migrate away from TeamCity and use 
> > something else. This could be an angle worth pursuing:
> > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-wi
> > th -unlimited-ci-cd-minutes-for-open-source
> > .
> >
> > In addition, someone mentioned other alternatives for open source 
> > projects, maybe whoever mentioned could chime in again.
> >
> > At this point having a build/test cycle going for master + all the 
> > PRs is the next big need to get this train moving.
> >
> >
> > Laimonas
> >
> > On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <si...@devhost.se>
> wrote:
> >
> >> Hi,
> >>
> >> I'm back after a week of districtions, with some further thoughts 
> >> about TeamCity.
> >>
> >> I don't think we can trust the agents to ever have a known state, 
> >> or even share a common state. The obvious(?) answer is docker. Is 
> >> docker installed on the agent, and are we allowed to pull down 
> >> docker images on the agents? Can we use windows containers? (I 
> >> believe it defaults to linux containers.)
> >>
> >> I believe that my permission level doesn't allow me to change the 
> >> existing build configuration, but I really hope there's a Docker 
> >> build step...
> >>
> >> The following docker run command will build the project. The %CD% 
> >> should expand to the lucenenet project root. I'm seeing some weird 
> >> timings, it takes minutes to restore packages for a single project, 
> >> but I'll be open and say that it could be something I've done to 
> >> myself in my experimenting with weird stuff. (And Windows started 
> >> informing me about low disk space when writing that sentence...)
> >>
> >> docker run ^
> >>  --rm ^
> >>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^  --volume %CD%:c:\build ^ 
> >> --workdir c:\build ^
> >>  mcr.microsoft.com/windows/servercore:1809 ^  POWERSHELL 
> >> "Import-Module .\build\psake.psm1; Invoke-psake
> >> .\build\build.ps1 -properties @{backup_files='false'}"
> >>
> >> Are there any support forums or similar for the hosted teamcity for 
> >> opensource projects? Or do we have any secret contacts that has all 
> >> the answers?
> >>
> >> // Simon Svensson
> >>
> >> On 2019-02-10 22:49, Simon Svensson wrote:
> >> > Hi,
> >> >
> >> > So, I messed something up and managed to build the master branch 
> >> > instead of my sisve-debugging-build branch... and it worked. I 
> >> > havn't figured out why. Probably server gnomes.
> >> >
> >> > It seems there are several agents/runners; and they aren't rolled 
> >> > back to a clean state between builds. I triggered the build on 
> >> > the real branch, and it got stuck on low disk space. After a few 
> >> > minutes I forcefully stopped it. It's still attempting to clean 
> >> > when I am writing this. (I wonder if I just messed up an 
> >> > agent...)
> >> >
> >> > The second attempt ran into the problem with non-working dotnet again.
> >> >
> >> > This is the build log from my accidentally working build. Note 
> >> > that there is already an old sdk installed, and a total of four 
> >> > sdks installed after dotnet-install has executed.
> >> >
> >> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=bui
> >> ld
> >> Log&_focus=1200
> >> >
> >> > It scares me slightly that this agent, the one that succeeded 
> >> > with the build, had builds since December 2016 in the build history...
> >> >
> >> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agen
> >> tH
> >> istory&page=31
> >> >
> >> > I will continue reading up on how TeamCity works, and the agents, 
> >> > and try to figure out why we had an agent with lots of sdks and 
> >> > .net runtimes installed.
> >> >
> >> > // Simon Svensson
> >> >
> >> > On 2019-02-09 19:41, Laimonas Simutis wrote:
> >> >> Simon,
> >> >>
> >> >> You should have access now to the project configuration. Let us 
> >> >> know
> >> how it
> >> >> goes. From the logs it looks like the project attempts to 
> >> >> install .net
> >> cli
> >> >> and fails and restore can't run, nor the build steps.
> >> >>
> >> >> Good luck. I will ping here if I have any breakthroughs myself.
> >> >>
> >> >>
> >> >> Laimonas
> >> >>
> >> >>
> >> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se>
> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> My JetBrains login would be the secretive "sisve@devhost.se".
> >> >>>
> >> >>> // Simon Svensson
> >> >>>
> >> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
> >> >>>> Simon,
> >> >>>>
> >> >>>> Oh my, I jumped into investigating the next builds in the 
> >> >>>> chain
> >> assuming
> >> >>>> that the green means all is well, but you are right. That did 
> >> >>>> not
> >> work.
> >> >>> Let
> >> >>>> me take a look at that and see what fails in base.
> >> >>>>
> >> >>>> What's your jetbrains login/email? I should be able to add you 
> >> >>>> to the administrative portion of the build
> >> >>>>
> >> >>>>
> >> >>>> Laimonas
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson 
> >> >>>> <si...@devhost.se>
> >> wrote:
> >> >>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> The main build has a green checkbox and the word "Success" 
> >> >>>>> next to
> >> it. I
> >> >>>>> would classify this as slightly inaccurate, the build totally
> >> failed.
> >> >>>>> The build script logged an error, but exited with a exit code 
> >> >>>>> of 0
> >> so no
> >> >>>>> error was signaled.
> >> >>>>>
> >> >>>>> In short, "dotnet.exe restore" failed for unknown reasons, 
> >> >>>>> and this continues in all builds since 1190. There are silent 
> >> >>>>> failures
> >> before the
> >> >>>>> restore; where the build scripts should output some 
> >> >>>>> information
> >> about
> >> >>>>> build numbers, directory paths and such, it should also 
> >> >>>>> output "dotnet.exe --version" and "dotnet.exe --info". These 
> >> >>>>> worked in
> >> build
> >> >>>>> 1189, but failed in 1190.
> >> >>>>>
> >> >>>>> A perhaps related observation is that we, starting with build 
> >> >>>>> 1190,
> >> also
> >> >>>>> need to install .NET SDK 2.0.0 every run. It could be 
> >> >>>>> relevant that
> >> it's
> >> >>>>> over 6 months between build 1189 and 1190, so there could 
> >> >>>>> perhaps
> >> be a
> >> >>>>> change in TeamCity we need to track down.
> >> >>>>>
> >> >>>>> Is there an administrative ui in TeamCity that I can get 
> >> >>>>> access
> to?
> >> I
> >> >>>>> believe the next step would be to debug (read: qualified
> >> >>>>> guessing)
> >> the
> >> >>>>> build script on a separate branch; would this be picked up
> >> automatically
> >> >>>>> by TeamCity?
> >> >>>>>
> >> >>>>> // Simon Svensson
> >> >>>>>
> >> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
> >> >>>>>> Simon,
> >> >>>>>>
> >> >>>>>> I was able to get the base builds going but as you already
> >> noticed, the
> >> >>>>>> subsequent builds are failing outright. The build structure 
> >> >>>>>> has
> >> changed
> >> >>>>>> quite a bit since the last time I as in there, so it's 
> >> >>>>>> taking me
> >> some
> >> >>>>> time
> >> >>>>>> to understand what's going on. I will continue to try to 
> >> >>>>>> untangle
> >> it or
> >> >>>>> set
> >> >>>>>> up a new build flow in there if that fails.
> >> >>>>>>
> >> >>>>>> Chris, here is the project link:
> >> >>>>>>
> >> >>>>>
> >> >>>
> >> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab
> >> =p
> >> rojectOverview
> >> >>>>>> if you want to take a look, perhaps you will see right away 
> >> >>>>>> what's
> >> up.
> >> >>>>>>
> >> >>>>>> Laimonas
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini 
> >> >>>>>> <ch...@brass9.com>
> >> >>> wrote:
> >> >>>>>>
> >> >>>>>>> We use Lucene.Net to do conventional search at:
> >> >>>>>>>
> >> >>>>>>> https://conservationx.com
> >> >>>>>>>
> >> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
> >> Levenshtein,
> >> >>> etc
> >> >>>>>>>
> >> >>>>>>> There was a problem mentioned about TeamCity; is there a 
> >> >>>>>>> way I can
> >> >>> learn
> >> >>>>>>> more in case I'd possibly be able to get it working? Where 
> >> >>>>>>> is it
> >> >>> checked
> >> >>>>>>> in, where is it running, etc
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >
>

Re: Update on dev happenings

Posted by Patric Forsgard <pa...@tasteful.se>.

I was notified today about Github Actions (
https://developer.github.com/actions/) that also can be executed on
developer computer with ACT (
https://github.com/nektos/act/blob/master/README.md). Can this be someting
to use instead? Yes, it will be a dependency on docker in that case to be
able to run everyting.

// Patric

On Thu, Aug 8, 2019, 06:37 Shad Storhaug <sh...@shadstorhaug.com> wrote:

> Update.
>
> I really didn't want to give up TeamCity because it is far better than
> Azure DevOps, but it seems that between the fact that JetBrains won't give
> us any more than one build agent, the fact it doesn't seem to be possible
> to get any Agents on Linux or Mac (at least none that will run powershell),
> and the fact that Microsoft broke the ability to get test results into
> TeamCity (I tried half a dozen configurations, but due to the other issues
> it didn't seem worth the effort) that Azure DevOps is a better choice. We
> definitely should switch to using Azure Artifacts regardless because of the
> generous amount of feed storage they allow (which frankly was why we didn't
> target .NET Standard 2.0 on all of the projects on the last release).
>
> We now have a YAML Azure DevOps pipeline in the repository. The test
> projects for Lucene.Net (core) were split into smaller pieces in order to
> run the tests in parallel in background jobs, and I was able to get the
> testing time down to about 40 minutes, which is a pretty safe margin less
> than the 1 hour limit Azure DevOps imposes on how long a job can run. I set
> it up so basically anyone can create a free Azure DevOps account, import
> the template, and it will automatically build, version a CI build based on
> a counter, and test all target frameworks on Windows, macOS and Ubuntu. Of
> course, it will build much faster if you setup the project as a public
> project so you can get 10 parallel jobs. If you enter an Azure Artifact
> feed ID (a GUID) as a variable, it will also push the packages and debug
> symbols into the feed (see the documentation in azure-pipelines.yml
> https://github.com/apache/lucenenet/blob/master/azure-pipelines.yml#L22).
>
> However, I have run into a couple of obstacles with actually getting this
> to work with the official repo. It seems that there are INFRA issues
> blocking it from working as a CI integration because Microsoft requires
> write repo access and Apache has a strict policy against it:
> https://issues.apache.org/jira/browse/INFRA-17030
>
> Furthermore, Microsoft doesn't support YAML in any "Other Git (generic)"
> repositories:
> https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/?view=azure-devops
> .
>
> Despite this, I was able to use the GitHub mirror of our repo by first
> setting it up on a fork, and then switching the GitHub repo name to
> apache/lucenenet, running in read-only mode. One problem solved.
>
> I have setup what I hope will be the official Lucene.NET organization
> here:
> https://dev.azure.com/lucene-net/Lucene.NET/_build/results?buildId=1&view=ms.vss-test-web.build-test-results-tab,
> but if we need to change it for any reason, nearly all of the configuration
> is in YAML so it isn't a lot of work.
>
> This configuration allows us to build a release because that is launched
> manually, but CI builds are another story. Azure DevOps doesn't get
> triggered by a push to the Apache repo, and they don't seem to have an
> option to poll for changes like we do in TeamCity. A couple of options I
> have been contemplating:
>
> 1. Setup a build in TeamCity to do an API call to Azure DevOps to launch
> the build:
> https://docs.microsoft.com/en-us/rest/api/azure/devops/build/?view=azure-devops-rest-5.0#queue-a-build
> 2. Setup an email address and register it with
> commits@lucenenet.apache.org and trigger the build whenever an email is
> received
>
> However, before I hack together a solution, I wanted to check whether any
> of the PMC know if it is possible to have INFRA setup a commit hook so we
> can do this in a more straightforward way? Possibly by setting up an Azure
> Function and having Apache call it via a URL:
> https://github.com/MicrosoftDocs/vsts-docs/issues/2366? I would ask them,
> but the docs say to ask here first...
>
>
> Thanks,
> Shad Storhaug (NightOwl888)
>
>
> -----Original Message-----
> From: Laimonas Simutis <la...@gmail.com>
> Sent: Tuesday, March 12, 2019 8:12 PM
> To: dev@lucenenet.apache.org
> Subject: Re: Update on dev happenings
>
> Quick update. I've had a successful build with azure pipelines, you can
> see it here:
> https://dev.azure.com/lsimutis/lsimutis/_build/results?buildId=5
>
> This is off of the fork of the current repo. Will see how the tests do
> there next and then perhaps we can seriously eval moving our builds off of
> teamcity on jetbrains. Or at the very least get the PRs building and
> verified automatically while we still decide if resurrecting teamcity setup
> is worth persuing.
>
>
> Laimonas
>
> On Tue, Feb 26, 2019 at 8:35 AM Laimonas Simutis <la...@gmail.com> wrote:
>
> > I wonder if the assumption that .net core needs to be installed is
> > outdated and we can find an agent with the .net core present.
> > psake abstraction here is hard to follow and reason what's going on...
> > I will check if there are other .net core based projects that are
> > being built in there that we can try to follow.
> >
> > Another option is for us to migrate away from TeamCity and use
> > something else. This could be an angle worth pursuing:
> > https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with
> > -unlimited-ci-cd-minutes-for-open-source
> > .
> >
> > In addition, someone mentioned other alternatives for open source
> > projects, maybe whoever mentioned could chime in again.
> >
> > At this point having a build/test cycle going for master + all the PRs
> > is the next big need to get this train moving.
> >
> >
> > Laimonas
> >
> > On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <si...@devhost.se>
> wrote:
> >
> >> Hi,
> >>
> >> I'm back after a week of districtions, with some further thoughts
> >> about TeamCity.
> >>
> >> I don't think we can trust the agents to ever have a known state, or
> >> even share a common state. The obvious(?) answer is docker. Is docker
> >> installed on the agent, and are we allowed to pull down docker images
> >> on the agents? Can we use windows containers? (I believe it defaults
> >> to linux containers.)
> >>
> >> I believe that my permission level doesn't allow me to change the
> >> existing build configuration, but I really hope there's a Docker
> >> build step...
> >>
> >> The following docker run command will build the project. The %CD%
> >> should expand to the lucenenet project root. I'm seeing some weird
> >> timings, it takes minutes to restore packages for a single project,
> >> but I'll be open and say that it could be something I've done to
> >> myself in my experimenting with weird stuff. (And Windows started
> >> informing me about low disk space when writing that sentence...)
> >>
> >> docker run ^
> >>  --rm ^
> >>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^  --volume %CD%:c:\build ^
> >> --workdir c:\build ^
> >>  mcr.microsoft.com/windows/servercore:1809 ^  POWERSHELL
> >> "Import-Module .\build\psake.psm1; Invoke-psake
> >> .\build\build.ps1 -properties @{backup_files='false'}"
> >>
> >> Are there any support forums or similar for the hosted teamcity for
> >> opensource projects? Or do we have any secret contacts that has all
> >> the answers?
> >>
> >> // Simon Svensson
> >>
> >> On 2019-02-10 22:49, Simon Svensson wrote:
> >> > Hi,
> >> >
> >> > So, I messed something up and managed to build the master branch
> >> > instead of my sisve-debugging-build branch... and it worked. I
> >> > havn't figured out why. Probably server gnomes.
> >> >
> >> > It seems there are several agents/runners; and they aren't rolled
> >> > back to a clean state between builds. I triggered the build on the
> >> > real branch, and it got stuck on low disk space. After a few
> >> > minutes I forcefully stopped it. It's still attempting to clean
> >> > when I am writing this. (I wonder if I just messed up an agent...)
> >> >
> >> > The second attempt ran into the problem with non-working dotnet again.
> >> >
> >> > This is the build log from my accidentally working build. Note that
> >> > there is already an old sdk installed, and a total of four sdks
> >> > installed after dotnet-install has executed.
> >> >
> >> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=build
> >> Log&_focus=1200
> >> >
> >> > It scares me slightly that this agent, the one that succeeded with
> >> > the build, had builds since December 2016 in the build history...
> >> >
> >> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentH
> >> istory&page=31
> >> >
> >> > I will continue reading up on how TeamCity works, and the agents,
> >> > and try to figure out why we had an agent with lots of sdks and
> >> > .net runtimes installed.
> >> >
> >> > // Simon Svensson
> >> >
> >> > On 2019-02-09 19:41, Laimonas Simutis wrote:
> >> >> Simon,
> >> >>
> >> >> You should have access now to the project configuration. Let us
> >> >> know
> >> how it
> >> >> goes. From the logs it looks like the project attempts to install
> >> >> .net
> >> cli
> >> >> and fails and restore can't run, nor the build steps.
> >> >>
> >> >> Good luck. I will ping here if I have any breakthroughs myself.
> >> >>
> >> >>
> >> >> Laimonas
> >> >>
> >> >>
> >> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se>
> >> wrote:
> >> >>
> >> >>> Hi,
> >> >>>
> >> >>> My JetBrains login would be the secretive "sisve@devhost.se".
> >> >>>
> >> >>> // Simon Svensson
> >> >>>
> >> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
> >> >>>> Simon,
> >> >>>>
> >> >>>> Oh my, I jumped into investigating the next builds in the chain
> >> assuming
> >> >>>> that the green means all is well, but you are right. That did
> >> >>>> not
> >> work.
> >> >>> Let
> >> >>>> me take a look at that and see what fails in base.
> >> >>>>
> >> >>>> What's your jetbrains login/email? I should be able to add you
> >> >>>> to the administrative portion of the build
> >> >>>>
> >> >>>>
> >> >>>> Laimonas
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se>
> >> wrote:
> >> >>>>
> >> >>>>> Hi,
> >> >>>>>
> >> >>>>> The main build has a green checkbox and the word "Success" next
> >> >>>>> to
> >> it. I
> >> >>>>> would classify this as slightly inaccurate, the build totally
> >> failed.
> >> >>>>> The build script logged an error, but exited with a exit code
> >> >>>>> of 0
> >> so no
> >> >>>>> error was signaled.
> >> >>>>>
> >> >>>>> In short, "dotnet.exe restore" failed for unknown reasons, and
> >> >>>>> this continues in all builds since 1190. There are silent
> >> >>>>> failures
> >> before the
> >> >>>>> restore; where the build scripts should output some information
> >> about
> >> >>>>> build numbers, directory paths and such, it should also output
> >> >>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked in
> >> build
> >> >>>>> 1189, but failed in 1190.
> >> >>>>>
> >> >>>>> A perhaps related observation is that we, starting with build
> >> >>>>> 1190,
> >> also
> >> >>>>> need to install .NET SDK 2.0.0 every run. It could be relevant
> >> >>>>> that
> >> it's
> >> >>>>> over 6 months between build 1189 and 1190, so there could
> >> >>>>> perhaps
> >> be a
> >> >>>>> change in TeamCity we need to track down.
> >> >>>>>
> >> >>>>> Is there an administrative ui in TeamCity that I can get access
> to?
> >> I
> >> >>>>> believe the next step would be to debug (read: qualified
> >> >>>>> guessing)
> >> the
> >> >>>>> build script on a separate branch; would this be picked up
> >> automatically
> >> >>>>> by TeamCity?
> >> >>>>>
> >> >>>>> // Simon Svensson
> >> >>>>>
> >> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
> >> >>>>>> Simon,
> >> >>>>>>
> >> >>>>>> I was able to get the base builds going but as you already
> >> noticed, the
> >> >>>>>> subsequent builds are failing outright. The build structure
> >> >>>>>> has
> >> changed
> >> >>>>>> quite a bit since the last time I as in there, so it's taking
> >> >>>>>> me
> >> some
> >> >>>>> time
> >> >>>>>> to understand what's going on. I will continue to try to
> >> >>>>>> untangle
> >> it or
> >> >>>>> set
> >> >>>>>> up a new build flow in there if that fails.
> >> >>>>>>
> >> >>>>>> Chris, here is the project link:
> >> >>>>>>
> >> >>>>>
> >> >>>
> >> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=p
> >> rojectOverview
> >> >>>>>> if you want to take a look, perhaps you will see right away
> >> >>>>>> what's
> >> up.
> >> >>>>>>
> >> >>>>>> Laimonas
> >> >>>>>>
> >> >>>>>>
> >> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini
> >> >>>>>> <ch...@brass9.com>
> >> >>> wrote:
> >> >>>>>>
> >> >>>>>>> We use Lucene.Net to do conventional search at:
> >> >>>>>>>
> >> >>>>>>> https://conservationx.com
> >> >>>>>>>
> >> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
> >> Levenshtein,
> >> >>> etc
> >> >>>>>>>
> >> >>>>>>> There was a problem mentioned about TeamCity; is there a way
> >> >>>>>>> I can
> >> >>> learn
> >> >>>>>>> more in case I'd possibly be able to get it working? Where is
> >> >>>>>>> it
> >> >>> checked
> >> >>>>>>> in, where is it running, etc
> >> >>>>>>>
> >> >>>>>>
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >>
> >
>

RE: Update on dev happenings

Posted by Shad Storhaug <sh...@shadstorhaug.com>.

Update.

I really didn't want to give up TeamCity because it is far better than Azure DevOps, but it seems that between the fact that JetBrains won't give us any more than one build agent, the fact it doesn't seem to be possible to get any Agents on Linux or Mac (at least none that will run powershell), and the fact that Microsoft broke the ability to get test results into TeamCity (I tried half a dozen configurations, but due to the other issues it didn't seem worth the effort) that Azure DevOps is a better choice. We definitely should switch to using Azure Artifacts regardless because of the generous amount of feed storage they allow (which frankly was why we didn't target .NET Standard 2.0 on all of the projects on the last release).

We now have a YAML Azure DevOps pipeline in the repository. The test projects for Lucene.Net (core) were split into smaller pieces in order to run the tests in parallel in background jobs, and I was able to get the testing time down to about 40 minutes, which is a pretty safe margin less than the 1 hour limit Azure DevOps imposes on how long a job can run. I set it up so basically anyone can create a free Azure DevOps account, import the template, and it will automatically build, version a CI build based on a counter, and test all target frameworks on Windows, macOS and Ubuntu. Of course, it will build much faster if you setup the project as a public project so you can get 10 parallel jobs. If you enter an Azure Artifact feed ID (a GUID) as a variable, it will also push the packages and debug symbols into the feed (see the documentation in azure-pipelines.yml https://github.com/apache/lucenenet/blob/master/azure-pipelines.yml#L22).

However, I have run into a couple of obstacles with actually getting this to work with the official repo. It seems that there are INFRA issues blocking it from working as a CI integration because Microsoft requires write repo access and Apache has a strict policy against it: https://issues.apache.org/jira/browse/INFRA-17030

Furthermore, Microsoft doesn't support YAML in any "Other Git (generic)" repositories: https://docs.microsoft.com/en-us/azure/devops/pipelines/repos/?view=azure-devops.

Despite this, I was able to use the GitHub mirror of our repo by first setting it up on a fork, and then switching the GitHub repo name to apache/lucenenet, running in read-only mode. One problem solved.

I have setup what I hope will be the official Lucene.NET organization here: https://dev.azure.com/lucene-net/Lucene.NET/_build/results?buildId=1&view=ms.vss-test-web.build-test-results-tab, but if we need to change it for any reason, nearly all of the configuration is in YAML so it isn't a lot of work.

This configuration allows us to build a release because that is launched manually, but CI builds are another story. Azure DevOps doesn't get triggered by a push to the Apache repo, and they don't seem to have an option to poll for changes like we do in TeamCity. A couple of options I have been contemplating:

1. Setup a build in TeamCity to do an API call to Azure DevOps to launch the build: https://docs.microsoft.com/en-us/rest/api/azure/devops/build/?view=azure-devops-rest-5.0#queue-a-build
2. Setup an email address and register it with commits@lucenenet.apache.org and trigger the build whenever an email is received

However, before I hack together a solution, I wanted to check whether any of the PMC know if it is possible to have INFRA setup a commit hook so we can do this in a more straightforward way? Possibly by setting up an Azure Function and having Apache call it via a URL: https://github.com/MicrosoftDocs/vsts-docs/issues/2366? I would ask them, but the docs say to ask here first...

Thanks,
Shad Storhaug (NightOwl888)

-----Original Message-----
From: Laimonas Simutis <la...@gmail.com> 
Sent: Tuesday, March 12, 2019 8:12 PM
To: dev@lucenenet.apache.org
Subject: Re: Update on dev happenings

Quick update. I've had a successful build with azure pipelines, you can see it here: https://dev.azure.com/lsimutis/lsimutis/_build/results?buildId=5

This is off of the fork of the current repo. Will see how the tests do there next and then perhaps we can seriously eval moving our builds off of teamcity on jetbrains. Or at the very least get the PRs building and verified automatically while we still decide if resurrecting teamcity setup is worth persuing.

Laimonas

On Tue, Feb 26, 2019 at 8:35 AM Laimonas Simutis <la...@gmail.com> wrote:

> I wonder if the assumption that .net core needs to be installed is 
> outdated and we can find an agent with the .net core present.
> psake abstraction here is hard to follow and reason what's going on... 
> I will check if there are other .net core based projects that are 
> being built in there that we can try to follow.
>
> Another option is for us to migrate away from TeamCity and use 
> something else. This could be an angle worth pursuing:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with
> -unlimited-ci-cd-minutes-for-open-source
> .
>
> In addition, someone mentioned other alternatives for open source 
> projects, maybe whoever mentioned could chime in again.
>
> At this point having a build/test cycle going for master + all the PRs 
> is the next big need to get this train moving.
>
>
> Laimonas
>
> On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <si...@devhost.se> wrote:
>
>> Hi,
>>
>> I'm back after a week of districtions, with some further thoughts 
>> about TeamCity.
>>
>> I don't think we can trust the agents to ever have a known state, or 
>> even share a common state. The obvious(?) answer is docker. Is docker 
>> installed on the agent, and are we allowed to pull down docker images 
>> on the agents? Can we use windows containers? (I believe it defaults 
>> to linux containers.)
>>
>> I believe that my permission level doesn't allow me to change the 
>> existing build configuration, but I really hope there's a Docker 
>> build step...
>>
>> The following docker run command will build the project. The %CD% 
>> should expand to the lucenenet project root. I'm seeing some weird 
>> timings, it takes minutes to restore packages for a single project, 
>> but I'll be open and say that it could be something I've done to 
>> myself in my experimenting with weird stuff. (And Windows started 
>> informing me about low disk space when writing that sentence...)
>>
>> docker run ^
>>  --rm ^
>>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^  --volume %CD%:c:\build ^  
>> --workdir c:\build ^
>>  mcr.microsoft.com/windows/servercore:1809 ^  POWERSHELL 
>> "Import-Module .\build\psake.psm1; Invoke-psake
>> .\build\build.ps1 -properties @{backup_files='false'}"
>>
>> Are there any support forums or similar for the hosted teamcity for 
>> opensource projects? Or do we have any secret contacts that has all 
>> the answers?
>>
>> // Simon Svensson
>>
>> On 2019-02-10 22:49, Simon Svensson wrote:
>> > Hi,
>> >
>> > So, I messed something up and managed to build the master branch 
>> > instead of my sisve-debugging-build branch... and it worked. I 
>> > havn't figured out why. Probably server gnomes.
>> >
>> > It seems there are several agents/runners; and they aren't rolled 
>> > back to a clean state between builds. I triggered the build on the 
>> > real branch, and it got stuck on low disk space. After a few 
>> > minutes I forcefully stopped it. It's still attempting to clean 
>> > when I am writing this. (I wonder if I just messed up an agent...)
>> >
>> > The second attempt ran into the problem with non-working dotnet again.
>> >
>> > This is the build log from my accidentally working build. Note that 
>> > there is already an old sdk installed, and a total of four sdks 
>> > installed after dotnet-install has executed.
>> >
>> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=build
>> Log&_focus=1200
>> >
>> > It scares me slightly that this agent, the one that succeeded with 
>> > the build, had builds since December 2016 in the build history...
>> >
>> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentH
>> istory&page=31
>> >
>> > I will continue reading up on how TeamCity works, and the agents, 
>> > and try to figure out why we had an agent with lots of sdks and 
>> > .net runtimes installed.
>> >
>> > // Simon Svensson
>> >
>> > On 2019-02-09 19:41, Laimonas Simutis wrote:
>> >> Simon,
>> >>
>> >> You should have access now to the project configuration. Let us 
>> >> know
>> how it
>> >> goes. From the logs it looks like the project attempts to install 
>> >> .net
>> cli
>> >> and fails and restore can't run, nor the build steps.
>> >>
>> >> Good luck. I will ping here if I have any breakthroughs myself.
>> >>
>> >>
>> >> Laimonas
>> >>
>> >>
>> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se>
>> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> My JetBrains login would be the secretive "sisve@devhost.se".
>> >>>
>> >>> // Simon Svensson
>> >>>
>> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
>> >>>> Simon,
>> >>>>
>> >>>> Oh my, I jumped into investigating the next builds in the chain
>> assuming
>> >>>> that the green means all is well, but you are right. That did 
>> >>>> not
>> work.
>> >>> Let
>> >>>> me take a look at that and see what fails in base.
>> >>>>
>> >>>> What's your jetbrains login/email? I should be able to add you 
>> >>>> to the administrative portion of the build
>> >>>>
>> >>>>
>> >>>> Laimonas
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se>
>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> The main build has a green checkbox and the word "Success" next 
>> >>>>> to
>> it. I
>> >>>>> would classify this as slightly inaccurate, the build totally
>> failed.
>> >>>>> The build script logged an error, but exited with a exit code 
>> >>>>> of 0
>> so no
>> >>>>> error was signaled.
>> >>>>>
>> >>>>> In short, "dotnet.exe restore" failed for unknown reasons, and 
>> >>>>> this continues in all builds since 1190. There are silent 
>> >>>>> failures
>> before the
>> >>>>> restore; where the build scripts should output some information
>> about
>> >>>>> build numbers, directory paths and such, it should also output 
>> >>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked in
>> build
>> >>>>> 1189, but failed in 1190.
>> >>>>>
>> >>>>> A perhaps related observation is that we, starting with build 
>> >>>>> 1190,
>> also
>> >>>>> need to install .NET SDK 2.0.0 every run. It could be relevant 
>> >>>>> that
>> it's
>> >>>>> over 6 months between build 1189 and 1190, so there could 
>> >>>>> perhaps
>> be a
>> >>>>> change in TeamCity we need to track down.
>> >>>>>
>> >>>>> Is there an administrative ui in TeamCity that I can get access to?
>> I
>> >>>>> believe the next step would be to debug (read: qualified 
>> >>>>> guessing)
>> the
>> >>>>> build script on a separate branch; would this be picked up
>> automatically
>> >>>>> by TeamCity?
>> >>>>>
>> >>>>> // Simon Svensson
>> >>>>>
>> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
>> >>>>>> Simon,
>> >>>>>>
>> >>>>>> I was able to get the base builds going but as you already
>> noticed, the
>> >>>>>> subsequent builds are failing outright. The build structure 
>> >>>>>> has
>> changed
>> >>>>>> quite a bit since the last time I as in there, so it's taking 
>> >>>>>> me
>> some
>> >>>>> time
>> >>>>>> to understand what's going on. I will continue to try to 
>> >>>>>> untangle
>> it or
>> >>>>> set
>> >>>>>> up a new build flow in there if that fails.
>> >>>>>>
>> >>>>>> Chris, here is the project link:
>> >>>>>>
>> >>>>>
>> >>>
>> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=p
>> rojectOverview
>> >>>>>> if you want to take a look, perhaps you will see right away 
>> >>>>>> what's
>> up.
>> >>>>>>
>> >>>>>> Laimonas
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini 
>> >>>>>> <ch...@brass9.com>
>> >>> wrote:
>> >>>>>>
>> >>>>>>> We use Lucene.Net to do conventional search at:
>> >>>>>>>
>> >>>>>>> https://conservationx.com
>> >>>>>>>
>> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
>> Levenshtein,
>> >>> etc
>> >>>>>>>
>> >>>>>>> There was a problem mentioned about TeamCity; is there a way 
>> >>>>>>> I can
>> >>> learn
>> >>>>>>> more in case I'd possibly be able to get it working? Where is 
>> >>>>>>> it
>> >>> checked
>> >>>>>>> in, where is it running, etc
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>

Re: Update on dev happenings

Posted by Laimonas Simutis <la...@gmail.com>.

Quick update. I've had a successful build with azure pipelines, you can see
it here: https://dev.azure.com/lsimutis/lsimutis/_build/results?buildId=5

This is off of the fork of the current repo. Will see how the tests do
there next and then perhaps we can seriously eval moving our builds off of
teamcity on jetbrains. Or at the very least get the PRs building and
verified automatically while we still decide if resurrecting teamcity setup
is worth persuing.


Laimonas

On Tue, Feb 26, 2019 at 8:35 AM Laimonas Simutis <la...@gmail.com> wrote:

> I wonder if the assumption that .net core needs to be installed is
> outdated and we can find an agent with the .net core present.
> psake abstraction here is hard to follow and reason what's going on... I
> will check if there are other .net core based projects that are being built
> in there that we can try to follow.
>
> Another option is for us to migrate away from TeamCity and use something
> else. This could be an angle worth pursuing:
> https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source
> .
>
> In addition, someone mentioned other alternatives for open source
> projects, maybe whoever mentioned could chime in again.
>
> At this point having a build/test cycle going for master + all the PRs is
> the next big need to get this train moving.
>
>
> Laimonas
>
> On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <si...@devhost.se> wrote:
>
>> Hi,
>>
>> I'm back after a week of districtions, with some further thoughts about
>> TeamCity.
>>
>> I don't think we can trust the agents to ever have a known state, or
>> even share a common state. The obvious(?) answer is docker. Is docker
>> installed on the agent, and are we allowed to pull down docker images on
>> the agents? Can we use windows containers? (I believe it defaults to
>> linux containers.)
>>
>> I believe that my permission level doesn't allow me to change the
>> existing build configuration, but I really hope there's a Docker build
>> step...
>>
>> The following docker run command will build the project. The %CD% should
>> expand to the lucenenet project root. I'm seeing some weird timings, it
>> takes minutes to restore packages for a single project, but I'll be open
>> and say that it could be something I've done to myself in my
>> experimenting with weird stuff. (And Windows started informing me about
>> low disk space when writing that sentence...)
>>
>> docker run ^
>>  --rm ^
>>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^
>>  --volume %CD%:c:\build ^
>>  --workdir c:\build ^
>>  mcr.microsoft.com/windows/servercore:1809 ^
>>  POWERSHELL "Import-Module .\build\psake.psm1; Invoke-psake
>> .\build\build.ps1 -properties @{backup_files='false'}"
>>
>> Are there any support forums or similar for the hosted teamcity for
>> opensource projects? Or do we have any secret contacts that has all the
>> answers?
>>
>> // Simon Svensson
>>
>> On 2019-02-10 22:49, Simon Svensson wrote:
>> > Hi,
>> >
>> > So, I messed something up and managed to build the master branch instead
>> > of my sisve-debugging-build branch... and it worked. I havn't figured
>> > out why. Probably server gnomes.
>> >
>> > It seems there are several agents/runners; and they aren't rolled back
>> > to a clean state between builds. I triggered the build on the real
>> > branch, and it got stuck on low disk space. After a few minutes I
>> > forcefully stopped it. It's still attempting to clean when I am writing
>> > this. (I wonder if I just messed up an agent...)
>> >
>> > The second attempt ran into the problem with non-working dotnet again.
>> >
>> > This is the build log from my accidentally working build. Note that
>> > there is already an old sdk installed, and a total of four sdks
>> > installed after dotnet-install has executed.
>> >
>> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=buildLog&_focus=1200
>> >
>> > It scares me slightly that this agent, the one that succeeded with the
>> > build, had builds since December 2016 in the build history...
>> >
>> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentHistory&page=31
>> >
>> > I will continue reading up on how TeamCity works, and the agents, and
>> > try to figure out why we had an agent with lots of sdks and .net
>> > runtimes installed.
>> >
>> > // Simon Svensson
>> >
>> > On 2019-02-09 19:41, Laimonas Simutis wrote:
>> >> Simon,
>> >>
>> >> You should have access now to the project configuration. Let us know
>> how it
>> >> goes. From the logs it looks like the project attempts to install .net
>> cli
>> >> and fails and restore can't run, nor the build steps.
>> >>
>> >> Good luck. I will ping here if I have any breakthroughs myself.
>> >>
>> >>
>> >> Laimonas
>> >>
>> >>
>> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se>
>> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> My JetBrains login would be the secretive "sisve@devhost.se".
>> >>>
>> >>> // Simon Svensson
>> >>>
>> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
>> >>>> Simon,
>> >>>>
>> >>>> Oh my, I jumped into investigating the next builds in the chain
>> assuming
>> >>>> that the green means all is well, but you are right. That did not
>> work.
>> >>> Let
>> >>>> me take a look at that and see what fails in base.
>> >>>>
>> >>>> What's your jetbrains login/email? I should be able to add you to the
>> >>>> administrative portion of the build
>> >>>>
>> >>>>
>> >>>> Laimonas
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se>
>> wrote:
>> >>>>
>> >>>>> Hi,
>> >>>>>
>> >>>>> The main build has a green checkbox and the word "Success" next to
>> it. I
>> >>>>> would classify this as slightly inaccurate, the build totally
>> failed.
>> >>>>> The build script logged an error, but exited with a exit code of 0
>> so no
>> >>>>> error was signaled.
>> >>>>>
>> >>>>> In short, "dotnet.exe restore" failed for unknown reasons, and this
>> >>>>> continues in all builds since 1190. There are silent failures
>> before the
>> >>>>> restore; where the build scripts should output some information
>> about
>> >>>>> build numbers, directory paths and such, it should also output
>> >>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked in
>> build
>> >>>>> 1189, but failed in 1190.
>> >>>>>
>> >>>>> A perhaps related observation is that we, starting with build 1190,
>> also
>> >>>>> need to install .NET SDK 2.0.0 every run. It could be relevant that
>> it's
>> >>>>> over 6 months between build 1189 and 1190, so there could perhaps
>> be a
>> >>>>> change in TeamCity we need to track down.
>> >>>>>
>> >>>>> Is there an administrative ui in TeamCity that I can get access to?
>> I
>> >>>>> believe the next step would be to debug (read: qualified guessing)
>> the
>> >>>>> build script on a separate branch; would this be picked up
>> automatically
>> >>>>> by TeamCity?
>> >>>>>
>> >>>>> // Simon Svensson
>> >>>>>
>> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
>> >>>>>> Simon,
>> >>>>>>
>> >>>>>> I was able to get the base builds going but as you already
>> noticed, the
>> >>>>>> subsequent builds are failing outright. The build structure has
>> changed
>> >>>>>> quite a bit since the last time I as in there, so it's taking me
>> some
>> >>>>> time
>> >>>>>> to understand what's going on. I will continue to try to untangle
>> it or
>> >>>>> set
>> >>>>>> up a new build flow in there if that fails.
>> >>>>>>
>> >>>>>> Chris, here is the project link:
>> >>>>>>
>> >>>>>
>> >>>
>> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
>> >>>>>> if you want to take a look, perhaps you will see right away what's
>> up.
>> >>>>>>
>> >>>>>> Laimonas
>> >>>>>>
>> >>>>>>
>> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com>
>> >>> wrote:
>> >>>>>>
>> >>>>>>> We use Lucene.Net to do conventional search at:
>> >>>>>>>
>> >>>>>>> https://conservationx.com
>> >>>>>>>
>> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
>> Levenshtein,
>> >>> etc
>> >>>>>>>
>> >>>>>>> There was a problem mentioned about TeamCity; is there a way I can
>> >>> learn
>> >>>>>>> more in case I'd possibly be able to get it working? Where is it
>> >>> checked
>> >>>>>>> in, where is it running, etc
>> >>>>>>>
>> >>>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>>
>

Re: Update on dev happenings

Posted by Laimonas Simutis <la...@gmail.com>.

I wonder if the assumption that .net core needs to be installed is outdated
and we can find an agent with the .net core present. psake abstraction here
is hard to follow and reason what's going on... I will check if there are
other .net core based projects that are being built in there that we can
try to follow.

Another option is for us to migrate away from TeamCity and use something
else. This could be an angle worth pursuing:
https://azure.microsoft.com/en-us/blog/announcing-azure-pipelines-with-unlimited-ci-cd-minutes-for-open-source
.

In addition, someone mentioned other alternatives for open source projects,
maybe whoever mentioned could chime in again.

At this point having a build/test cycle going for master + all the PRs is
the next big need to get this train moving.


Laimonas

On Sun, Feb 17, 2019 at 12:25 PM Simon Svensson <si...@devhost.se> wrote:

> Hi,
>
> I'm back after a week of districtions, with some further thoughts about
> TeamCity.
>
> I don't think we can trust the agents to ever have a known state, or
> even share a common state. The obvious(?) answer is docker. Is docker
> installed on the agent, and are we allowed to pull down docker images on
> the agents? Can we use windows containers? (I believe it defaults to
> linux containers.)
>
> I believe that my permission level doesn't allow me to change the
> existing build configuration, but I really hope there's a Docker build
> step...
>
> The following docker run command will build the project. The %CD% should
> expand to the lucenenet project root. I'm seeing some weird timings, it
> takes minutes to restore packages for a single project, but I'll be open
> and say that it could be something I've done to myself in my
> experimenting with weird stuff. (And Windows started informing me about
> low disk space when writing that sentence...)
>
> docker run ^
>  --rm ^
>  --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^
>  --volume %CD%:c:\build ^
>  --workdir c:\build ^
>  mcr.microsoft.com/windows/servercore:1809 ^
>  POWERSHELL "Import-Module .\build\psake.psm1; Invoke-psake
> .\build\build.ps1 -properties @{backup_files='false'}"
>
> Are there any support forums or similar for the hosted teamcity for
> opensource projects? Or do we have any secret contacts that has all the
> answers?
>
> // Simon Svensson
>
> On 2019-02-10 22:49, Simon Svensson wrote:
> > Hi,
> >
> > So, I messed something up and managed to build the master branch instead
> > of my sisve-debugging-build branch... and it worked. I havn't figured
> > out why. Probably server gnomes.
> >
> > It seems there are several agents/runners; and they aren't rolled back
> > to a clean state between builds. I triggered the build on the real
> > branch, and it got stuck on low disk space. After a few minutes I
> > forcefully stopped it. It's still attempting to clean when I am writing
> > this. (I wonder if I just messed up an agent...)
> >
> > The second attempt ran into the problem with non-working dotnet again.
> >
> > This is the build log from my accidentally working build. Note that
> > there is already an old sdk installed, and a total of four sdks
> > installed after dotnet-install has executed.
> >
> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=buildLog&_focus=1200
> >
> > It scares me slightly that this agent, the one that succeeded with the
> > build, had builds since December 2016 in the build history...
> >
> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentHistory&page=31
> >
> > I will continue reading up on how TeamCity works, and the agents, and
> > try to figure out why we had an agent with lots of sdks and .net
> > runtimes installed.
> >
> > // Simon Svensson
> >
> > On 2019-02-09 19:41, Laimonas Simutis wrote:
> >> Simon,
> >>
> >> You should have access now to the project configuration. Let us know
> how it
> >> goes. From the logs it looks like the project attempts to install .net
> cli
> >> and fails and restore can't run, nor the build steps.
> >>
> >> Good luck. I will ping here if I have any breakthroughs myself.
> >>
> >>
> >> Laimonas
> >>
> >>
> >> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se> wrote:
> >>
> >>> Hi,
> >>>
> >>> My JetBrains login would be the secretive "sisve@devhost.se".
> >>>
> >>> // Simon Svensson
> >>>
> >>> On 2019-02-08 17:26, Laimonas Simutis wrote:
> >>>> Simon,
> >>>>
> >>>> Oh my, I jumped into investigating the next builds in the chain
> assuming
> >>>> that the green means all is well, but you are right. That did not
> work.
> >>> Let
> >>>> me take a look at that and see what fails in base.
> >>>>
> >>>> What's your jetbrains login/email? I should be able to add you to the
> >>>> administrative portion of the build
> >>>>
> >>>>
> >>>> Laimonas
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se>
> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> The main build has a green checkbox and the word "Success" next to
> it. I
> >>>>> would classify this as slightly inaccurate, the build totally failed.
> >>>>> The build script logged an error, but exited with a exit code of 0
> so no
> >>>>> error was signaled.
> >>>>>
> >>>>> In short, "dotnet.exe restore" failed for unknown reasons, and this
> >>>>> continues in all builds since 1190. There are silent failures before
> the
> >>>>> restore; where the build scripts should output some information about
> >>>>> build numbers, directory paths and such, it should also output
> >>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked in build
> >>>>> 1189, but failed in 1190.
> >>>>>
> >>>>> A perhaps related observation is that we, starting with build 1190,
> also
> >>>>> need to install .NET SDK 2.0.0 every run. It could be relevant that
> it's
> >>>>> over 6 months between build 1189 and 1190, so there could perhaps be
> a
> >>>>> change in TeamCity we need to track down.
> >>>>>
> >>>>> Is there an administrative ui in TeamCity that I can get access to? I
> >>>>> believe the next step would be to debug (read: qualified guessing)
> the
> >>>>> build script on a separate branch; would this be picked up
> automatically
> >>>>> by TeamCity?
> >>>>>
> >>>>> // Simon Svensson
> >>>>>
> >>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
> >>>>>> Simon,
> >>>>>>
> >>>>>> I was able to get the base builds going but as you already noticed,
> the
> >>>>>> subsequent builds are failing outright. The build structure has
> changed
> >>>>>> quite a bit since the last time I as in there, so it's taking me
> some
> >>>>> time
> >>>>>> to understand what's going on. I will continue to try to untangle
> it or
> >>>>> set
> >>>>>> up a new build flow in there if that fails.
> >>>>>>
> >>>>>> Chris, here is the project link:
> >>>>>>
> >>>>>
> >>>
> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
> >>>>>> if you want to take a look, perhaps you will see right away what's
> up.
> >>>>>>
> >>>>>> Laimonas
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com>
> >>> wrote:
> >>>>>>
> >>>>>>> We use Lucene.Net to do conventional search at:
> >>>>>>>
> >>>>>>> https://conservationx.com
> >>>>>>>
> >>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch,
> Levenshtein,
> >>> etc
> >>>>>>>
> >>>>>>> There was a problem mentioned about TeamCity; is there a way I can
> >>> learn
> >>>>>>> more in case I'd possibly be able to get it working? Where is it
> >>> checked
> >>>>>>> in, where is it running, etc
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>

Re: Update on dev happenings

Posted by Simon Svensson <si...@devhost.se>.

Hi,

I'm back after a week of districtions, with some further thoughts about
TeamCity.

I don't think we can trust the agents to ever have a known state, or
even share a common state. The obvious(?) answer is docker. Is docker
installed on the agent, and are we allowed to pull down docker images on
the agents? Can we use windows containers? (I believe it defaults to
linux containers.)

I believe that my permission level doesn't allow me to change the
existing build configuration, but I really hope there's a Docker build
step...

The following docker run command will build the project. The %CD% should
expand to the lucenenet project root. I'm seeing some weird timings, it
takes minutes to restore packages for a single project, but I'll be open
and say that it could be something I've done to myself in my
experimenting with weird stuff. (And Windows started informing me about
low disk space when writing that sentence...)

docker run ^
 --rm ^
 --env DOTNET_CLI_TELEMETRY_OPTOUT=1 ^
 --volume %CD%:c:\build ^
 --workdir c:\build ^
 mcr.microsoft.com/windows/servercore:1809 ^
 POWERSHELL "Import-Module .\build\psake.psm1; Invoke-psake
.\build\build.ps1 -properties @{backup_files='false'}"

Are there any support forums or similar for the hosted teamcity for
opensource projects? Or do we have any secret contacts that has all the
answers?

// Simon Svensson

On 2019-02-10 22:49, Simon Svensson wrote:
> Hi,
> 
> So, I messed something up and managed to build the master branch instead
> of my sisve-debugging-build branch... and it worked. I havn't figured
> out why. Probably server gnomes.
> 
> It seems there are several agents/runners; and they aren't rolled back
> to a clean state between builds. I triggered the build on the real
> branch, and it got stuck on low disk space. After a few minutes I
> forcefully stopped it. It's still attempting to clean when I am writing
> this. (I wonder if I just messed up an agent...)
> 
> The second attempt ran into the problem with non-working dotnet again.
> 
> This is the build log from my accidentally working build. Note that
> there is already an old sdk installed, and a total of four sdks
> installed after dotnet-install has executed.
> https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=buildLog&_focus=1200
> 
> It scares me slightly that this agent, the one that succeeded with the
> build, had builds since December 2016 in the build history...
> https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentHistory&page=31
> 
> I will continue reading up on how TeamCity works, and the agents, and
> try to figure out why we had an agent with lots of sdks and .net
> runtimes installed.
> 
> // Simon Svensson
> 
> On 2019-02-09 19:41, Laimonas Simutis wrote:
>> Simon,
>>
>> You should have access now to the project configuration. Let us know how it
>> goes. From the logs it looks like the project attempts to install .net cli
>> and fails and restore can't run, nor the build steps.
>>
>> Good luck. I will ping here if I have any breakthroughs myself.
>>
>>
>> Laimonas
>>
>>
>> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se> wrote:
>>
>>> Hi,
>>>
>>> My JetBrains login would be the secretive "sisve@devhost.se".
>>>
>>> // Simon Svensson
>>>
>>> On 2019-02-08 17:26, Laimonas Simutis wrote:
>>>> Simon,
>>>>
>>>> Oh my, I jumped into investigating the next builds in the chain assuming
>>>> that the green means all is well, but you are right. That did not work.
>>> Let
>>>> me take a look at that and see what fails in base.
>>>>
>>>> What's your jetbrains login/email? I should be able to add you to the
>>>> administrative portion of the build
>>>>
>>>>
>>>> Laimonas
>>>>
>>>>
>>>>
>>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> The main build has a green checkbox and the word "Success" next to it. I
>>>>> would classify this as slightly inaccurate, the build totally failed.
>>>>> The build script logged an error, but exited with a exit code of 0 so no
>>>>> error was signaled.
>>>>>
>>>>> In short, "dotnet.exe restore" failed for unknown reasons, and this
>>>>> continues in all builds since 1190. There are silent failures before the
>>>>> restore; where the build scripts should output some information about
>>>>> build numbers, directory paths and such, it should also output
>>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked in build
>>>>> 1189, but failed in 1190.
>>>>>
>>>>> A perhaps related observation is that we, starting with build 1190, also
>>>>> need to install .NET SDK 2.0.0 every run. It could be relevant that it's
>>>>> over 6 months between build 1189 and 1190, so there could perhaps be a
>>>>> change in TeamCity we need to track down.
>>>>>
>>>>> Is there an administrative ui in TeamCity that I can get access to? I
>>>>> believe the next step would be to debug (read: qualified guessing) the
>>>>> build script on a separate branch; would this be picked up automatically
>>>>> by TeamCity?
>>>>>
>>>>> // Simon Svensson
>>>>>
>>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
>>>>>> Simon,
>>>>>>
>>>>>> I was able to get the base builds going but as you already noticed, the
>>>>>> subsequent builds are failing outright. The build structure has changed
>>>>>> quite a bit since the last time I as in there, so it's taking me some
>>>>> time
>>>>>> to understand what's going on. I will continue to try to untangle it or
>>>>> set
>>>>>> up a new build flow in there if that fails.
>>>>>>
>>>>>> Chris, here is the project link:
>>>>>>
>>>>>
>>> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
>>>>>> if you want to take a look, perhaps you will see right away what's up.
>>>>>>
>>>>>> Laimonas
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com>
>>> wrote:
>>>>>>
>>>>>>> We use Lucene.Net to do conventional search at:
>>>>>>>
>>>>>>> https://conservationx.com
>>>>>>>
>>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein,
>>> etc
>>>>>>>
>>>>>>> There was a problem mentioned about TeamCity; is there a way I can
>>> learn
>>>>>>> more in case I'd possibly be able to get it working? Where is it
>>> checked
>>>>>>> in, where is it running, etc
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>

Re: Update on dev happenings

Posted by Simon Svensson <si...@devhost.se>.

Hi,

So, I messed something up and managed to build the master branch instead
of my sisve-debugging-build branch... and it worked. I havn't figured
out why. Probably server gnomes.

It seems there are several agents/runners; and they aren't rolled back
to a clean state between builds. I triggered the build on the real
branch, and it got stuck on low disk space. After a few minutes I
forcefully stopped it. It's still attempting to clean when I am writing
this. (I wonder if I just messed up an agent...)

The second attempt ran into the problem with non-working dotnet again.

This is the build log from my accidentally working build. Note that
there is already an old sdk installed, and a total of four sdks
installed after dotnet-install has executed.
https://teamcity.jetbrains.com/viewLog.html?buildId=1960125&tab=buildLog&_focus=1200

It scares me slightly that this agent, the one that succeeded with the
build, had builds since December 2016 in the build history...
https://teamcity.jetbrains.com/agentDetails.html?id=170591&tab=agentHistory&page=31

I will continue reading up on how TeamCity works, and the agents, and
try to figure out why we had an agent with lots of sdks and .net
runtimes installed.

// Simon Svensson

On 2019-02-09 19:41, Laimonas Simutis wrote:
> Simon,
> 
> You should have access now to the project configuration. Let us know how it
> goes. From the logs it looks like the project attempts to install .net cli
> and fails and restore can't run, nor the build steps.
> 
> Good luck. I will ping here if I have any breakthroughs myself.
> 
> 
> Laimonas
> 
> 
> On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se> wrote:
> 
>> Hi,
>>
>> My JetBrains login would be the secretive "sisve@devhost.se".
>>
>> // Simon Svensson
>>
>> On 2019-02-08 17:26, Laimonas Simutis wrote:
>>> Simon,
>>>
>>> Oh my, I jumped into investigating the next builds in the chain assuming
>>> that the green means all is well, but you are right. That did not work.
>> Let
>>> me take a look at that and see what fails in base.
>>>
>>> What's your jetbrains login/email? I should be able to add you to the
>>> administrative portion of the build
>>>
>>>
>>> Laimonas
>>>
>>>
>>>
>>> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se> wrote:
>>>
>>>> Hi,
>>>>
>>>> The main build has a green checkbox and the word "Success" next to it. I
>>>> would classify this as slightly inaccurate, the build totally failed.
>>>> The build script logged an error, but exited with a exit code of 0 so no
>>>> error was signaled.
>>>>
>>>> In short, "dotnet.exe restore" failed for unknown reasons, and this
>>>> continues in all builds since 1190. There are silent failures before the
>>>> restore; where the build scripts should output some information about
>>>> build numbers, directory paths and such, it should also output
>>>> "dotnet.exe --version" and "dotnet.exe --info". These worked in build
>>>> 1189, but failed in 1190.
>>>>
>>>> A perhaps related observation is that we, starting with build 1190, also
>>>> need to install .NET SDK 2.0.0 every run. It could be relevant that it's
>>>> over 6 months between build 1189 and 1190, so there could perhaps be a
>>>> change in TeamCity we need to track down.
>>>>
>>>> Is there an administrative ui in TeamCity that I can get access to? I
>>>> believe the next step would be to debug (read: qualified guessing) the
>>>> build script on a separate branch; would this be picked up automatically
>>>> by TeamCity?
>>>>
>>>> // Simon Svensson
>>>>
>>>> On 2019-02-06 13:57, Laimonas Simutis wrote:
>>>>> Simon,
>>>>>
>>>>> I was able to get the base builds going but as you already noticed, the
>>>>> subsequent builds are failing outright. The build structure has changed
>>>>> quite a bit since the last time I as in there, so it's taking me some
>>>> time
>>>>> to understand what's going on. I will continue to try to untangle it or
>>>> set
>>>>> up a new build flow in there if that fails.
>>>>>
>>>>> Chris, here is the project link:
>>>>>
>>>>
>> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
>>>>> if you want to take a look, perhaps you will see right away what's up.
>>>>>
>>>>> Laimonas
>>>>>
>>>>>
>>>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com>
>> wrote:
>>>>>
>>>>>> We use Lucene.Net to do conventional search at:
>>>>>>
>>>>>> https://conservationx.com
>>>>>>
>>>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein,
>> etc
>>>>>>
>>>>>> There was a problem mentioned about TeamCity; is there a way I can
>> learn
>>>>>> more in case I'd possibly be able to get it working? Where is it
>> checked
>>>>>> in, where is it running, etc
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Update on dev happenings

Posted by Laimonas Simutis <la...@gmail.com>.

Simon,

You should have access now to the project configuration. Let us know how it
goes. From the logs it looks like the project attempts to install .net cli
and fails and restore can't run, nor the build steps.

Good luck. I will ping here if I have any breakthroughs myself.


Laimonas


On Sat, Feb 9, 2019 at 3:52 AM Simon Svensson <si...@devhost.se> wrote:

> Hi,
>
> My JetBrains login would be the secretive "sisve@devhost.se".
>
> // Simon Svensson
>
> On 2019-02-08 17:26, Laimonas Simutis wrote:
> > Simon,
> >
> > Oh my, I jumped into investigating the next builds in the chain assuming
> > that the green means all is well, but you are right. That did not work.
> Let
> > me take a look at that and see what fails in base.
> >
> > What's your jetbrains login/email? I should be able to add you to the
> > administrative portion of the build
> >
> >
> > Laimonas
> >
> >
> >
> > On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se> wrote:
> >
> >> Hi,
> >>
> >> The main build has a green checkbox and the word "Success" next to it. I
> >> would classify this as slightly inaccurate, the build totally failed.
> >> The build script logged an error, but exited with a exit code of 0 so no
> >> error was signaled.
> >>
> >> In short, "dotnet.exe restore" failed for unknown reasons, and this
> >> continues in all builds since 1190. There are silent failures before the
> >> restore; where the build scripts should output some information about
> >> build numbers, directory paths and such, it should also output
> >> "dotnet.exe --version" and "dotnet.exe --info". These worked in build
> >> 1189, but failed in 1190.
> >>
> >> A perhaps related observation is that we, starting with build 1190, also
> >> need to install .NET SDK 2.0.0 every run. It could be relevant that it's
> >> over 6 months between build 1189 and 1190, so there could perhaps be a
> >> change in TeamCity we need to track down.
> >>
> >> Is there an administrative ui in TeamCity that I can get access to? I
> >> believe the next step would be to debug (read: qualified guessing) the
> >> build script on a separate branch; would this be picked up automatically
> >> by TeamCity?
> >>
> >> // Simon Svensson
> >>
> >> On 2019-02-06 13:57, Laimonas Simutis wrote:
> >>> Simon,
> >>>
> >>> I was able to get the base builds going but as you already noticed, the
> >>> subsequent builds are failing outright. The build structure has changed
> >>> quite a bit since the last time I as in there, so it's taking me some
> >> time
> >>> to understand what's going on. I will continue to try to untangle it or
> >> set
> >>> up a new build flow in there if that fails.
> >>>
> >>> Chris, here is the project link:
> >>>
> >>
> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
> >>> if you want to take a look, perhaps you will see right away what's up.
> >>>
> >>> Laimonas
> >>>
> >>>
> >>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com>
> wrote:
> >>>
> >>>> We use Lucene.Net to do conventional search at:
> >>>>
> >>>> https://conservationx.com
> >>>>
> >>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein,
> etc
> >>>>
> >>>> There was a problem mentioned about TeamCity; is there a way I can
> learn
> >>>> more in case I'd possibly be able to get it working? Where is it
> checked
> >>>> in, where is it running, etc
> >>>>
> >>>
> >>
> >
>

Re: Update on dev happenings

Posted by Simon Svensson <si...@devhost.se>.

Hi,

My JetBrains login would be the secretive "sisve@devhost.se".

// Simon Svensson

On 2019-02-08 17:26, Laimonas Simutis wrote:
> Simon,
> 
> Oh my, I jumped into investigating the next builds in the chain assuming
> that the green means all is well, but you are right. That did not work. Let
> me take a look at that and see what fails in base.
> 
> What's your jetbrains login/email? I should be able to add you to the
> administrative portion of the build
> 
> 
> Laimonas
> 
> 
> 
> On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se> wrote:
> 
>> Hi,
>>
>> The main build has a green checkbox and the word "Success" next to it. I
>> would classify this as slightly inaccurate, the build totally failed.
>> The build script logged an error, but exited with a exit code of 0 so no
>> error was signaled.
>>
>> In short, "dotnet.exe restore" failed for unknown reasons, and this
>> continues in all builds since 1190. There are silent failures before the
>> restore; where the build scripts should output some information about
>> build numbers, directory paths and such, it should also output
>> "dotnet.exe --version" and "dotnet.exe --info". These worked in build
>> 1189, but failed in 1190.
>>
>> A perhaps related observation is that we, starting with build 1190, also
>> need to install .NET SDK 2.0.0 every run. It could be relevant that it's
>> over 6 months between build 1189 and 1190, so there could perhaps be a
>> change in TeamCity we need to track down.
>>
>> Is there an administrative ui in TeamCity that I can get access to? I
>> believe the next step would be to debug (read: qualified guessing) the
>> build script on a separate branch; would this be picked up automatically
>> by TeamCity?
>>
>> // Simon Svensson
>>
>> On 2019-02-06 13:57, Laimonas Simutis wrote:
>>> Simon,
>>>
>>> I was able to get the base builds going but as you already noticed, the
>>> subsequent builds are failing outright. The build structure has changed
>>> quite a bit since the last time I as in there, so it's taking me some
>> time
>>> to understand what's going on. I will continue to try to untangle it or
>> set
>>> up a new build flow in there if that fails.
>>>
>>> Chris, here is the project link:
>>>
>> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
>>> if you want to take a look, perhaps you will see right away what's up.
>>>
>>> Laimonas
>>>
>>>
>>> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com> wrote:
>>>
>>>> We use Lucene.Net to do conventional search at:
>>>>
>>>> https://conservationx.com
>>>>
>>>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein, etc
>>>>
>>>> There was a problem mentioned about TeamCity; is there a way I can learn
>>>> more in case I'd possibly be able to get it working? Where is it checked
>>>> in, where is it running, etc
>>>>
>>>
>>
>

Re: Update on dev happenings

Posted by Laimonas Simutis <la...@gmail.com>.

Simon,

Oh my, I jumped into investigating the next builds in the chain assuming
that the green means all is well, but you are right. That did not work. Let
me take a look at that and see what fails in base.

What's your jetbrains login/email? I should be able to add you to the
administrative portion of the build


Laimonas



On Fri, Feb 8, 2019 at 1:01 AM Simon Svensson <si...@devhost.se> wrote:

> Hi,
>
> The main build has a green checkbox and the word "Success" next to it. I
> would classify this as slightly inaccurate, the build totally failed.
> The build script logged an error, but exited with a exit code of 0 so no
> error was signaled.
>
> In short, "dotnet.exe restore" failed for unknown reasons, and this
> continues in all builds since 1190. There are silent failures before the
> restore; where the build scripts should output some information about
> build numbers, directory paths and such, it should also output
> "dotnet.exe --version" and "dotnet.exe --info". These worked in build
> 1189, but failed in 1190.
>
> A perhaps related observation is that we, starting with build 1190, also
> need to install .NET SDK 2.0.0 every run. It could be relevant that it's
> over 6 months between build 1189 and 1190, so there could perhaps be a
> change in TeamCity we need to track down.
>
> Is there an administrative ui in TeamCity that I can get access to? I
> believe the next step would be to debug (read: qualified guessing) the
> build script on a separate branch; would this be picked up automatically
> by TeamCity?
>
> // Simon Svensson
>
> On 2019-02-06 13:57, Laimonas Simutis wrote:
> > Simon,
> >
> > I was able to get the base builds going but as you already noticed, the
> > subsequent builds are failing outright. The build structure has changed
> > quite a bit since the last time I as in there, so it's taking me some
> time
> > to understand what's going on. I will continue to try to untangle it or
> set
> > up a new build flow in there if that fails.
> >
> > Chris, here is the project link:
> >
> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
> > if you want to take a look, perhaps you will see right away what's up.
> >
> > Laimonas
> >
> >
> > On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com> wrote:
> >
> >> We use Lucene.Net to do conventional search at:
> >>
> >> https://conservationx.com
> >>
> >> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein, etc
> >>
> >> There was a problem mentioned about TeamCity; is there a way I can learn
> >> more in case I'd possibly be able to get it working? Where is it checked
> >> in, where is it running, etc
> >>
> >
>

Re: Update on dev happenings

Posted by Simon Svensson <si...@devhost.se>.

Hi,

The main build has a green checkbox and the word "Success" next to it. I
would classify this as slightly inaccurate, the build totally failed.
The build script logged an error, but exited with a exit code of 0 so no
error was signaled.

In short, "dotnet.exe restore" failed for unknown reasons, and this
continues in all builds since 1190. There are silent failures before the
restore; where the build scripts should output some information about
build numbers, directory paths and such, it should also output
"dotnet.exe --version" and "dotnet.exe --info". These worked in build
1189, but failed in 1190.

A perhaps related observation is that we, starting with build 1190, also
need to install .NET SDK 2.0.0 every run. It could be relevant that it's
over 6 months between build 1189 and 1190, so there could perhaps be a
change in TeamCity we need to track down.

Is there an administrative ui in TeamCity that I can get access to? I
believe the next step would be to debug (read: qualified guessing) the
build script on a separate branch; would this be picked up automatically
by TeamCity?

// Simon Svensson

On 2019-02-06 13:57, Laimonas Simutis wrote:
> Simon,
> 
> I was able to get the base builds going but as you already noticed, the
> subsequent builds are failing outright. The build structure has changed
> quite a bit since the last time I as in there, so it's taking me some time
> to understand what's going on. I will continue to try to untangle it or set
> up a new build flow in there if that fails.
> 
> Chris, here is the project link:
> https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
> if you want to take a look, perhaps you will see right away what's up.
> 
> Laimonas
> 
> 
> On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com> wrote:
> 
>> We use Lucene.Net to do conventional search at:
>>
>> https://conservationx.com
>>
>> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein, etc
>>
>> There was a problem mentioned about TeamCity; is there a way I can learn
>> more in case I'd possibly be able to get it working? Where is it checked
>> in, where is it running, etc
>>
>

Re: Update on dev happenings

Posted by Laimonas Simutis <la...@gmail.com>.

Simon,

I was able to get the base builds going but as you already noticed, the
subsequent builds are failing outright. The build structure has changed
quite a bit since the last time I as in there, so it's taking me some time
to understand what's going on. I will continue to try to untangle it or set
up a new build flow in there if that fails.

Chris, here is the project link:
https://teamcity.jetbrains.com/project.html?projectId=LuceneNet&tab=projectOverview
if you want to take a look, perhaps you will see right away what's up.

Laimonas

On Wed, Feb 6, 2019 at 12:46 AM Chris Moschini <ch...@brass9.com> wrote:

> We use Lucene.Net to do conventional search at:
>
> https://conservationx.com
>
> Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein, etc
>
> There was a problem mentioned about TeamCity; is there a way I can learn
> more in case I'd possibly be able to get it working? Where is it checked
> in, where is it running, etc
>

Re: Update on dev happenings

Posted by Chris Moschini <ch...@brass9.com>.

We use Lucene.Net to do conventional search at:

https://conservationx.com

Analyzer, Indexer, Highlighter, Tokenizer, FuzzySearch, Levenshtein, etc

There was a problem mentioned about TeamCity; is there a way I can learn
more in case I'd possibly be able to get it working? Where is it checked
in, where is it running, etc

RE: Update on dev happenings

Posted by Aaron Meyers <Aa...@microsoft.com.INVALID>.

I have no opposition to moving forward with 4.8 release personally, but I would like to just call out that while we (Microsoft Power BI) are using Lucene.Net 4.8 beta5 in both shipping product and service production environments, we are using a fairly narrow slice of Lucene.Net functionality (just the core index with our own analyzers; none of the query parsers, document ranking or additional modules outside of the Lucene.Net assembly). So I wouldn't put all our confidence in the stability of 4.8 across the board on my team's usage.

If anyone else is currently using 4.8 for a more typical search scenario it would be nice to hear :)

-----Original Message-----
From: Andrey Kurdumov <ka...@googlemail.com.INVALID> 
Sent: Tuesday, February 5, 2019 9:18 PM
To: dev@lucenenet.apache.org
Subject: Re: Update on dev happenings

I mean “use 4.8 beta” not “you 4.8 beta”. Was too excited to spellcheck, my apologizes On Wed, Feb 6, 2019 at 11:16 Andrey Kurdumov <ka...@googlemail.com>
wrote:

> As an observer which you 4.8 beta on the project I excited to hear 
> this news On Tue, Feb 5, 2019 at 22:54 Elad Margalit 
> <el...@gmail.com> wrote:
>
>> Completely agree, the version should be released, done > perfect.
>>
>> On Tue, Feb 5, 2019 at 6:42 PM Prescott Nasser 
>> <ge...@hotmail.com>
>> wrote:
>>
>> > Sounds pretty logical to me
>> > ________________________________
>> > From: Chris Moschini <ch...@brass9.com>
>> > Sent: Tuesday, February 5, 2019 5:12:57 PM
>> > To: dev@lucenenet.apache.org
>> > Subject: Re: Update on dev happenings
>> >
>> > Given the length of time since 4.8 became stable and the fact no 
>> > one is working on the ICU stuff, I think it would be best to move 
>> > forward with releasing 4.8 with a Known Issues addendum that 
>> > encourages others to contribute if they'd like things like a complete ICU.
>> >
>> > Lucene itself has advanced several major versions in the meantime, 
>> > to
>> 7.6,
>> > and catching up with them to reach better API parity seems like a 
>> > better use of any core dev resources than trying to perfect 
>> > something 3 major versions old.
>> >
>>
>

Re: Update on dev happenings

Posted by Andrey Kurdumov <ka...@googlemail.com.INVALID>.

I mean “use 4.8 beta” not “you 4.8 beta”. Was too excited to spellcheck, my
apologizes
On Wed, Feb 6, 2019 at 11:16 Andrey Kurdumov <ka...@googlemail.com>
wrote:

> As an observer which you 4.8 beta on the project I excited to hear this
> news
> On Tue, Feb 5, 2019 at 22:54 Elad Margalit <el...@gmail.com> wrote:
>
>> Completely agree, the version should be released, done > perfect.
>>
>> On Tue, Feb 5, 2019 at 6:42 PM Prescott Nasser <ge...@hotmail.com>
>> wrote:
>>
>> > Sounds pretty logical to me
>> > ________________________________
>> > From: Chris Moschini <ch...@brass9.com>
>> > Sent: Tuesday, February 5, 2019 5:12:57 PM
>> > To: dev@lucenenet.apache.org
>> > Subject: Re: Update on dev happenings
>> >
>> > Given the length of time since 4.8 became stable and the fact no one is
>> > working on the ICU stuff, I think it would be best to move forward with
>> > releasing 4.8 with a Known Issues addendum that encourages others to
>> > contribute if they'd like things like a complete ICU.
>> >
>> > Lucene itself has advanced several major versions in the meantime, to
>> 7.6,
>> > and catching up with them to reach better API parity seems like a better
>> > use of any core dev resources than trying to perfect something 3 major
>> > versions old.
>> >
>>
>

Re: Update on dev happenings

Posted by Andrey Kurdumov <ka...@googlemail.com.INVALID>.

As an observer which you 4.8 beta on the project I excited to hear this news
On Tue, Feb 5, 2019 at 22:54 Elad Margalit <el...@gmail.com> wrote:

> Completely agree, the version should be released, done > perfect.
>
> On Tue, Feb 5, 2019 at 6:42 PM Prescott Nasser <ge...@hotmail.com>
> wrote:
>
> > Sounds pretty logical to me
> > ________________________________
> > From: Chris Moschini <ch...@brass9.com>
> > Sent: Tuesday, February 5, 2019 5:12:57 PM
> > To: dev@lucenenet.apache.org
> > Subject: Re: Update on dev happenings
> >
> > Given the length of time since 4.8 became stable and the fact no one is
> > working on the ICU stuff, I think it would be best to move forward with
> > releasing 4.8 with a Known Issues addendum that encourages others to
> > contribute if they'd like things like a complete ICU.
> >
> > Lucene itself has advanced several major versions in the meantime, to
> 7.6,
> > and catching up with them to reach better API parity seems like a better
> > use of any core dev resources than trying to perfect something 3 major
> > versions old.
> >
>

Re: Update on dev happenings

Posted by Elad Margalit <el...@gmail.com>.

Completely agree, the version should be released, done > perfect.

On Tue, Feb 5, 2019 at 6:42 PM Prescott Nasser <ge...@hotmail.com>
wrote:

> Sounds pretty logical to me
> ________________________________
> From: Chris Moschini <ch...@brass9.com>
> Sent: Tuesday, February 5, 2019 5:12:57 PM
> To: dev@lucenenet.apache.org
> Subject: Re: Update on dev happenings
>
> Given the length of time since 4.8 became stable and the fact no one is
> working on the ICU stuff, I think it would be best to move forward with
> releasing 4.8 with a Known Issues addendum that encourages others to
> contribute if they'd like things like a complete ICU.
>
> Lucene itself has advanced several major versions in the meantime, to 7.6,
> and catching up with them to reach better API parity seems like a better
> use of any core dev resources than trying to perfect something 3 major
> versions old.
>

Re: Update on dev happenings

Posted by Prescott Nasser <ge...@hotmail.com>.

Sounds pretty logical to me
________________________________
From: Chris Moschini <ch...@brass9.com>
Sent: Tuesday, February 5, 2019 5:12:57 PM
To: dev@lucenenet.apache.org
Subject: Re: Update on dev happenings

Given the length of time since 4.8 became stable and the fact no one is
working on the ICU stuff, I think it would be best to move forward with
releasing 4.8 with a Known Issues addendum that encourages others to
contribute if they'd like things like a complete ICU.

Lucene itself has advanced several major versions in the meantime, to 7.6,
and catching up with them to reach better API parity seems like a better
use of any core dev resources than trying to perfect something 3 major
versions old.

Re: Update on dev happenings

Posted by Chris Moschini <ch...@brass9.com>.

Given the length of time since 4.8 became stable and the fact no one is
working on the ICU stuff, I think it would be best to move forward with
releasing 4.8 with a Known Issues addendum that encourages others to
contribute if they'd like things like a complete ICU.

Lucene itself has advanced several major versions in the meantime, to 7.6,
and catching up with them to reach better API parity seems like a better
use of any core dev resources than trying to perfect something 3 major
versions old.

Re: Update on dev happenings

Posted by Simon Svensson <si...@devhost.se>.

Hi,

Are there any news about the TeamCity installation? I don't speak
TeamCityish, but I would presume the build step is not producing the
artifacts that the tests step require.

> Failed to resolve artifacts from <teamcity.codebetter.com / Lucene.NET
/ Portable Builds / Lucene.Net Base Build, build #4.8.0-ci0000001192 [id
1920179]>

// Simon Svensson

On 2019-01-25 14:14, Laimonas Simutis wrote:
> Hello there,
> 
> A quick note on what was discovered/done this past week while getting
> things back up and running.
> 
> - Simon's fix for https://issues.apache.org/jira/browse/LUCENENET-597 was
> merged
> - lucenenet-site repo is up where we will commit the website code
> - old/irrelevant JIRA tickets cleaned up (more cleanup coming)
> 
> Next:
> 
> - working on getting the TeamCity builds going so that we could verify with
> more confidence another of Simon's fixes (
> https://github.com/apache/lucenenet/pull/218), and in general get the
> feedback from builds flowing to have more confidence on the 4.8 stability.
> - work with Shannon to get his PR for the website merged into the site
> repo, hopefully, we hear from him soon.
> - clean up CONTRIBUTE.md so it's very clear what sort of next tasks are
> outstanding
> 
> Also, if you have time, take a look at the comment Shad made in this ticket
> (comment highlighted):
> https://issues.apache.org/jira/browse/LUCENENET-601?focusedCommentId=16465234.
> I think that's a very good overview of where we are at with 4.8. Short
> version: if we decide to go without fully ICU working bits, we could
> potentially release 4.8. But that's a discussion for a separate thread.
> 
> 
> Have a great Friday everyone,
> 
> Laimonas
>