Re: convenience artifacts are signed and uploaded

[Konstantin Boudnik <co...@apache.org>]

Just to wrap it up here as well. As reported on http://bit.ly/1XiYRRM - the
situation has been resolved and the guilty part is RPM's long-standing
inability to deal with subkeys.

Cos

On Mon, Aug 31, 2015 at 12:57AM, Konstantin Boudnik wrote:
> Arghhh... of course. I completely forgot about this ;( And I have updated the
> wiki release page, to avoid forgetting this in the future.
> 
> Synced the repodata folders to S3 - everything should be fine now.
> Thanks a lot for your help!
>   Cos
> 
> On Mon, Aug 31, 2015 at 03:37AM, Evans Ye wrote:
> > Hi Cos,
> > 
> > I finally found the issue by reproducing a repo in my S3 buckets.
> > The answer is that the repodata needs to be regenerated after RPMs are
> > being signed.
> > 
> > Here's my testing steps:
> > 
> > 1. download the whole built repo from jenkins release built as archive.zip
> > and unzip it
> > 2. sign RPMs:
> >   $ find -name *.rpm | xargs rpm --define="%_gpg_name Evans Ye" --addsign
> > *3. recreate the metadata*
> > *  $ rm -rf repodata; createrepo -v .*
> > 3. upload to S3 bucket:
> >   $
> > aws s3 cp bigtop-1.0.0-rpm-centos-6-output/
> > s3://evansye/bigtop/centos/6/x86_64 --recursive
> > 
> > My repo:
> > [bigtop]
> > name=Bigtop
> > enabled=1
> > gpgcheck=1
> > baseurl=https://s3-us-west-1.amazonaws.com/evansye/bigtop/centos/6/x86_64
> > gpgkey=https://s3-us-west-1.amazonaws.com/evansye/bigtop/KEYS
> > 
> > Using my repo, the armored pgpkey will be automatically imported and the
> > packages are being installed without a problem.
> > 
> > Here's the TODO summary for out official repos:
> > 1. Recreate metadata for yum repos
> > 2. Replace http://archive.apache.org/dist/bigtop/KEYS to an armored key
> > 
> > I think the bigtop official repo is critical to our user experience and I
> > hope we can get it fixed ASAP. To be mentioned that Jazz Yao-Tsung Wang hits
> > the issue too and reported back to me as well.
> > If you're kind of busy I can help, too. Just let me know. Thanks!
> > 
> > Evans
> > 
> > 2015-08-24 1:48 GMT+08:00 Evans Ye <ev...@apache.org>:
> > 
> > > Hey Cos, thanks for updating the packages.
> > > I was testing the repos these days, but it still behaving strange.
> > > I constantly got "no more mirrors to try" when yum installing rpms from S3
> > > repo:
> > >
> > > Error Downloading Packages:
> > >   bigtop-utils-1.0.0-1.el6.noarch: failure:
> > > bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm from bigtop: [Errno
> > > 256] No more mirrors to try.
> > >
> > > However, the package exist and has been signed as well:
> > >
> > > $ wget
> > > http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm
> > > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm
> > > bigtop-utils-1.0.0-1.el6.noarch.rpm: RSA sha1 ((MD5) PGP) md5 NOT OK
> > > (MISSING KEYS: (MD5) PGP#d0c3824f)
> > >
> > > I can't figure out why for days, so I just what to bring up the issue and
> > > see if you have any clue.
> > > Does it work on your end?
> > >
> > >
> > > 2015-08-21 11:32 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >
> > >> Ok, my mistake - I have only signed src rpms ;( My bad - apologies for
> > >> wasting
> > >> your time. Now, with the key - I can really figure out what's going on
> > >> here.
> > >> The key seems to be fine, yet rpm --checksig isn't happy about it. I've
> > >> re-imported the key as you suggested, yet ir complains that the key is
> > >> missing
> > >> during the validation phase. I want to get to the bottom of it, but at
> > >> least
> > >> we have the packages signed now (the upload should be over in about an
> > >> hour, I
> > >> hope).
> > >>
> > >> Thanks,
> > >>   Cos
> > >>
> > >> On Thu, Aug 20, 2015 at 01:31AM, Evans Ye wrote:
> > >> > Hi Cos,
> > >> >
> > >> > I've verified deb repos, and they're good as what you said, however, it
> > >> > seems that rpms are still not being signed. I don't have too much
> > >> knowledge
> > >> > on this so I did my homework and conduct the following evaluation.
> > >> > Here's my evaluation steps, please advise if any thing incorrect:
> > >> >
> > >> > First, download a rpm from S3, which should be already signed:
> > >> >
> > >> > $ wget
> > >> >
> > >> http://bigtop.s3.amazonaws.com/releases/1.0.0/centos/6/x86_64/bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm
> > >> > $ rpm --checksig bigtop-utils/noarch/bigtop-utils-1.0.0-1.el6.noarch.rpm
> > >> > bigtop-utils-1.0.0-1.el6.noarch.rpm.1: sha1 md5 OK
> > >> >
> > >> > As shown above, the rpm does not being signed.
> > >> > A signed rpm should be looked like this:
> > >> >
> > >> > $ rpm --checksig puppetlabs-release-el-6.noarch.rpm
> > >> > puppetlabs-release-el-6.noarch.rpm: rsa sha1 (md5) pgp md5 OK
> > >> >
> > >> > And after I signed the rpm by my key, the rpm looks good now:
> > >> >
> > >> > $ rpm --addsign bigtop-utils-1.0.0-1.el6.noarch.rpm
> > >> > $ rpm --checksig bigtop-utils-1.0.0-1.el6.noarch.rpm
> > >> > bigtop-utils-1.0.0-1.el6.noarch.rpm: rsa sha1 (md5) pgp md5 OK
> > >> >
> > >> > OTOH, it looks like the pgpkey for rpm packages needs to be armored when
> > >> > exporting, for example:
> > >> >
> > >> > $ gpg --armor --output KEYS --export 'Evans Ye'
> > >> >
> > >> > Otherwise, an error occurs when importing a non-armored key:
> > >> >
> > >> > $ rpm --import https://dist.apache.org/repos/dist/release/bigtop/KEYS
> > >> > error: https://dist.apache.org/repos/dist/release/bigtop/KEYS: key 2
> > >> not an
> > >> > armored public key.
> > >> >
> > >> > A good thing for that is we can fix it in the cloud.
> > >> > Sorry for not discovering this at the very beginning. :(
> > >> >
> > >> > Evans
> > >> >
> > >> >
> > >> > 2015-08-19 4:12 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >> >
> > >> > > On Wed, Aug 19, 2015 at 02:12AM, Evans Ye wrote:
> > >> > > > Well, sorry. When I do manually test on those repos, I got package
> > >> is not
> > >> > > > signed message...
> > >> > > > It turns out that our puppet recipe sets pgpkey checking to false,
> > >> which
> > >> > > is
> > >> > > > why I got things working without a problem.
> > >> > > > I assume the repo should just work by dropping it into
> > >> /etc/yum.repos.d/.
> > >> > > > No need to import pgpkey manually, right?
> > >> > >
> > >> > > Hmm... with ubuntu repo the signatures are there because I had to run
> > >> > > apt-addkey manually to make apt recognize the signatures. Otherwise,
> > >> > > apt-get update didn't work.
> > >> > >
> > >> > > You might need to do something similar with yum - I am not really
> > >> sure.
> > >> > > But I
> > >> > > am positive that I have signed the packages per the insructions on our
> > >> > > release
> > >> > > page.
> > >> > >
> > >> > >   Cos
> > >> > >
> > >> > > > 2015-08-18 2:39 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >> > > >
> > >> > > > > I think we are in good position - I was worries about keys being
> > >> not
> > >> > > found
> > >> > > > > or
> > >> > > > > something like this. If two of them are ok, then the rest should
> > >> be
> > >> > > fine
> > >> > > > > too.
> > >> > > > >
> > >> > > > > Thank you very much for the confirmation and testing - really
> > >> > > appreciate
> > >> > > > > it!
> > >> > > > > I will send the announcement shortly.
> > >> > > > >
> > >> > > > > Regards,
> > >> > > > >   Cos
> > >> > > > >
> > >> > > > > On Tue, Aug 18, 2015 at 02:05AM, Evans Ye wrote:
> > >> > > > > > We're so close to the finished line! Thank you everybody!
> > >> > > > > > I took sometime to run the deployment test before I go to bed.
> > >> > > > > > Specifically I've tested centos-6, debian-8 repo, both work
> > >> like a
> > >> > > charm.
> > >> > > > > > It should be all good, if no hurry I'll do more test tomorrow.
> > >> :)
> > >> > > > > >
> > >> > > > > > 2015-08-17 14:00 GMT+08:00 Konstantin Boudnik <co...@apache.org>:
> > >> > > > > >
> > >> > > > > > > See? I can not even write word "dense" without making a
> > >> typo...
> > >> > > That's
> > >> > > > > how
> > >> > > > > > > hot
> > >> > > > > > > it is. Anyway, I have uploaded all signed packages to s3 (oh
> > >> boy,
> > >> > > was
> > >> > > > > it
> > >> > > > > > > fun?), and updated the repo files in the release under
> > >> > > > > > >
> > >> > > https://dist.apache.org/repos/dist/release/bigtop/bigtop-1.0.0/
> > >> > > > > > >
> > >> > > > > > > If you have a cycle or two - please do some quick validation
> > >> and I
> > >> > > will
> > >> > > > > > > send
> > >> > > > > > > the announcement to make the release final.
> > >> > > > > > >
> > >> > > > > > > Thanks everyone
> > >> > > > > > >   Cos
> > >> > > > > > >
> > >> > > > > > > On Mon, Aug 17, 2015 at 03:18AM, Konstantin Boudnik wrote:
> > >> > > > > > > > Ah, of cource ... I am so sense when it's  in the 100's F
> > >> > > outside ;(
> > >> > > > > > > Thanks!
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Sun, Aug 16, 2015 at 12:17PM, Roman Shaposhnik wrote:
> > >> > > > > > > > > On Sat, Aug 15, 2015 at 11:27 PM, Konstantin Boudnik <
> > >> > > > > cos@apache.org>
> > >> > > > > > > wrote:
> > >> > > > > > > > > > Make sense - wgetting the stuff now. centos is done,
> > >> debs and
> > >> > > > > fedora
> > >> > > > > > > to go.
> > >> > > > > > > > > > Weirdly, deb repos are huge >2.5GB for whatever reason.
> > >> > > > > > > > >
> > >> > > > > > > > > IIRC, this had to do with double caching of packages.
> > >> IOW, you
> > >> > > only
> > >> > > > > > > > > need to make this available:
> > >> > > > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >> http://bigtop01.cloudera.org:8080/view/Releases/job/Bigtop-1.0.0-deb/BUILD_ENVIRONMENTS=debian-8,label=docker-slave-07/lastSuccessfulBuild/artifact/output/apt/
> > >> > > > > > > > >
> > >> > > > > > > > > not the top level dir.
> > >> > > > > > > > >
> > >> > > > > > > > > Thanks,
> > >> > > > > > > > > Roman.
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > >
> > >> > >
> > >>
> > >
> > >