Fwd: Build Lucene/Solr in cloud

[David Smiley <ds...@apache.org>]

FYI Relevant conversation about Crave.io from 2.5 years ago.  I chat with
"Uv" (Yuvraaj), Crave's CEO, on occasion about this arrangement.

I arranged for "crave run" to do the needful without all the extra params
needed.  I just did a build a minute ago -- took 6m 27sec on branch_9_1;
one failing test.  And my old machine was unphased because it wasn't doing
the work :-). It's nice to kick off such a build in the background and be
productive doing development / whatever while it runs.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


---------- Forwarded message ---------
From: Yuvraaj Kelkar <yu...@gmail.com>
Date: Thu, May 21, 2020 at 12:32 AM
Subject: Re: Build Lucene/Solr in cloud
To: dev@lucene.apache.org <de...@lucene.apache.org>
Cc: Solr/Lucene Dev <de...@lucene.apache.org>


Hi David,

@node spec:
The current setup has Crave configured to spin up a 16 core (VCPU) with 64
GB ram.
I can change that to more cores, more memory if you want to try it out.

@variable number of arguments:
Crave *does* accept a variable number of arguments. It is very similar to
the ssh style of unix commands:
crave run -- command with multiple parameters -with -flags -like -this

You could even run multiple commands:
crave run 'command1 args1 ; command2 args2'

@laptop sleep:
There's two ways to survive sleep:

   1. Start it in "detached" mode: Starts the task in the background on the
   remote node. You don't need a persistent network connection for this. It
   just runs it in the background. Use: crave run --detached -- command args

   2. Start Crave in foreground mode, then let the laptop sleep. This will
   cause the network connection to break and the crave client will terminate.
   However, the task on the remote end will continue. Client side failures
   don't stop remote side work.

Note: A Ctrl+C on the crave client side will be transmitted to the remote
end and can be used to terminate a foreground task.
To see running tasks, use crave list .
To kill background tasks use crave stop .

Thanks,
-Uv

On May 20 2020, at 7:09 pm, David Smiley <da...@gmail.com> wrote:

I don't know what Crave.io does should I, say, close my laptop and go to
sleep and come back to it.  If it could survive that somehow then that'd
be a sweet feature!  I doubt my simple rsync script plays well with that so
I don't dare.

~ David


[image: Sent from Mailspring]
On Wed, May 20, 2020 at 10:05 PM David Smiley <da...@gmail.com>
wrote:

Hi Yuvraaj,

I do builds on a corporate provided VM that I don't know a lot about but it
reports 16 CPUs and plenty of RAM (32GB?).  I tuned the Solr build to use
10 test runners, which seems to work out best.  Lately, the Solr-only tests
take about 21-22 minutes or so.  I run with "ant -f solr/build.xml test"

Since I didn't need to touch Crave's config, I don't have much feedback for
it.  It'd be nice if you could pass it a variable number of args that it
would run similar to an "ssh" command, etc.  My buildbox.sh script (linked
in the gist) works this way.

When I next do dev from my personal laptop, I'll use Crave.io.

~ David

On Wed, May 20, 2020 at 7:39 PM Yuvraaj Kelkar <yu...@gmail.com> wrote:

Hi Pushkar,
Thanks for your kind words!

Hi David,
Glad to hear the build was smooth with Crave.  I echo Pushkar's questions.
Also, I am all ears to any other feedback and questions you may have.

Thanks,
-Uv

On May 20 2020, at 5:17 am, Pushkar Raste <pu...@gmail.com> wrote:

Hi David,
Good to hear it worked for you. I suggested Crave team to join the mailing
list so they can directly hear the feedback and answer any questions. For
rest of us in the community who may not have beefier boxes can you share
what is config of your buildbox, how long builds take on the buildbox and
if there are any tricks you have to make builds run faster. I don't think
Crave supports gradle builds for Lucene/Solr yet but the Crave team can add
it if needed (IIUC gradle build is work in progress so not sure how many of
us are using it). Crave team will use your feedback to reconfigure the
cloud server.

Note: I don't work for Crave but know the founder/CEO and he was
generous enough to help set up crave to build Lucene/Solr. I thought others
in the community can benefit from it as well.


On Tue, May 19, 2020 at 11:40 AM David Smiley <da...@gmail.com>
wrote:

Hi,

This is pretty cool!  It worked for me right away without issue.  I have my
own similar rsync based script I've been using to build Lucene/Solr on
other machines/VM --
https://gist.github.com/dsmiley/fdd589758cd74009222c518640b093b5  It's
generous for crave.io to offer free build servers.  However most of my
builds I will continue to use my "buildbox" script because I have access to
a much beefier machine.

~ David


On Mon, May 18, 2020 at 10:00 AM Pushkar Raste <pu...@gmail.com>
wrote:

Hi,

Building Lucene/Solr with all tests takes about ~50 minutes to an hour
depending on how powerful your machine is.
Try out https://crave.io/
<https://link.getmailspring.com/link/7D34916F-D0D9-4C76-8E3B-246153A48881@getmailspring.com/0?redirect=https%3A%2F%2Fcrave.io%2F&recipient=cHVzaGthci5yYXN0ZUBnbWFpbC5jb20%3D>
to
run your builds in the cloud and free up resources on your development
machine.

To run the builds in the cloud, just download crave and simply run
following command from within your lucene/solr source code directory:

$ <path/to/crave>/crave run ant <target>

Crave will pick up the local changes on your development machine while
building in the cloud.

Let me know your experience.