You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by ramkrishna vasudevan <ra...@gmail.com> on 2013/06/20 12:16:15 UTC
Minor versions for HFile and HFileBlock
Minor versions are basically for the HFileBlock format. So any change that
we try to do in the HFileBlock format we need to increase the minor version.
The FixedFileTrailer has the minor version 3 written in it.
static final int MAX_MINOR_VERSION = 3.
This forms the min version that is written to the FixedFileTrailer of the
HFile.
Inside the HFileBlock constructor
{code}
if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
this.checksumType = b.get();
this.bytesPerChecksum = b.getInt();
this.onDiskDataSizeWithHeader = b.getInt();
} else {
this.checksumType = ChecksumType.NULL.getCode();
this.bytesPerChecksum = 0;
this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
}
{code}
where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and minorVersion =
MAX_MINOR_VERSION.
So if i need to make some changes to the HFileBlock format adding some
additional bytes to the header, i need to increase the minorversion number.
But since the minor version is already 3 i need to introduce another
constant which is greater than 3
and then add the new bytes in the header.
Something like make
static final int MAX_MINOR_VERSION = 4.
Introduce a NEW_CONSTANT =4 in the HFileBlock
if(minorVersion >= NEW_CONSTANT)
{
//Add new bytes.
}
where minorVersion = MAX_MINOR_VERSION .
So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the new
one that i introduce now goes to 4. What i was thinking is increasing by 1
everywhere in the code should help but seeing the code that is not the
case.
So is this approach valid or am i missing something? Testing with this
approach am able to make an existing version of Trunk to work with new
changes that i had made.
Regards
Ram
Re: Minor versions for HFile and HFileBlock
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Thanks for the comments. So i hope the approach that am following is
correct.
Regards
Ram
On Thu, Jun 20, 2013 at 9:45 PM, Andrew Purtell <ap...@apache.org> wrote:
> +1 to consolidating these constants into one place (HFile.java?).
>
>
>
>
> On Thu, Jun 20, 2013 at 7:25 AM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > The FixedFileTrailer uses MAX_MINOR_VERSION.
> MINOR_VERSION_WITH_FAKED_KEY
> > is being checked to use the INDEX_MAGIC but not sure where it is written
> to
> > the trailer.
> > Some of the Minor versions are spread across the code i felt. Will check
> on
> > that too.
> >
> > Regards
> > Ram
> >
> >
> > On Thu, Jun 20, 2013 at 5:10 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > Take a look at:
> > >
> > >
> >
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> > >
> > > static final int MAX_MINOR_VERSION = 3;
> > >
> > > /** Minor versions starting with this number have faked index key */
> > > static final int MINOR_VERSION_WITH_FAKED_KEY = 3;
> > >
> > > I think the new constant should be added in HFileReaderV2.java so that
> it
> > > is easier to observe all the changes to minor version in one place.
> > >
> > > Cheers
> > >
> > > On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
> > > ramkrishna.s.vasudevan@gmail.com> wrote:
> > >
> > > > Minor versions are basically for the HFileBlock format. So any
> change
> > > that
> > > > we try to do in the HFileBlock format we need to increase the minor
> > > > version.
> > > >
> > > > The FixedFileTrailer has the minor version 3 written in it.
> > > > static final int MAX_MINOR_VERSION = 3.
> > > > This forms the min version that is written to the FixedFileTrailer of
> > the
> > > > HFile.
> > > >
> > > > Inside the HFileBlock constructor
> > > > {code}
> > > > if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
> > > > this.checksumType = b.get();
> > > > this.bytesPerChecksum = b.getInt();
> > > > this.onDiskDataSizeWithHeader = b.getInt();
> > > > } else {
> > > > this.checksumType = ChecksumType.NULL.getCode();
> > > > this.bytesPerChecksum = 0;
> > > > this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
> > > >
> > > > HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
> > > > }
> > > > {code}
> > > > where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and
> minorVersion
> > > =
> > > > MAX_MINOR_VERSION.
> > > >
> > > > So if i need to make some changes to the HFileBlock format adding
> some
> > > > additional bytes to the header, i need to increase the minorversion
> > > number.
> > > > But since the minor version is already 3 i need to introduce another
> > > > constant which is greater than 3
> > > > and then add the new bytes in the header.
> > > > Something like make
> > > > static final int MAX_MINOR_VERSION = 4.
> > > > Introduce a NEW_CONSTANT =4 in the HFileBlock
> > > > if(minorVersion >= NEW_CONSTANT)
> > > > {
> > > > //Add new bytes.
> > > > }
> > > > where minorVersion = MAX_MINOR_VERSION .
> > > > So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the
> > new
> > > > one that i introduce now goes to 4. What i was thinking is increasing
> > by
> > > 1
> > > > everywhere in the code should help but seeing the code that is not
> the
> > > > case.
> > > > So is this approach valid or am i missing something? Testing with
> this
> > > > approach am able to make an existing version of Trunk to work with
> new
> > > > changes that i had made.
> > > >
> > > > Regards
> > > > Ram
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Re: Minor versions for HFile and HFileBlock
Posted by Andrew Purtell <ap...@apache.org>.
+1 to consolidating these constants into one place (HFile.java?).
On Thu, Jun 20, 2013 at 7:25 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> The FixedFileTrailer uses MAX_MINOR_VERSION. MINOR_VERSION_WITH_FAKED_KEY
> is being checked to use the INDEX_MAGIC but not sure where it is written to
> the trailer.
> Some of the Minor versions are spread across the code i felt. Will check on
> that too.
>
> Regards
> Ram
>
>
> On Thu, Jun 20, 2013 at 5:10 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > Take a look at:
> >
> >
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> >
> > static final int MAX_MINOR_VERSION = 3;
> >
> > /** Minor versions starting with this number have faked index key */
> > static final int MINOR_VERSION_WITH_FAKED_KEY = 3;
> >
> > I think the new constant should be added in HFileReaderV2.java so that it
> > is easier to observe all the changes to minor version in one place.
> >
> > Cheers
> >
> > On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com> wrote:
> >
> > > Minor versions are basically for the HFileBlock format. So any change
> > that
> > > we try to do in the HFileBlock format we need to increase the minor
> > > version.
> > >
> > > The FixedFileTrailer has the minor version 3 written in it.
> > > static final int MAX_MINOR_VERSION = 3.
> > > This forms the min version that is written to the FixedFileTrailer of
> the
> > > HFile.
> > >
> > > Inside the HFileBlock constructor
> > > {code}
> > > if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
> > > this.checksumType = b.get();
> > > this.bytesPerChecksum = b.getInt();
> > > this.onDiskDataSizeWithHeader = b.getInt();
> > > } else {
> > > this.checksumType = ChecksumType.NULL.getCode();
> > > this.bytesPerChecksum = 0;
> > > this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
> > >
> > > HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
> > > }
> > > {code}
> > > where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and minorVersion
> > =
> > > MAX_MINOR_VERSION.
> > >
> > > So if i need to make some changes to the HFileBlock format adding some
> > > additional bytes to the header, i need to increase the minorversion
> > number.
> > > But since the minor version is already 3 i need to introduce another
> > > constant which is greater than 3
> > > and then add the new bytes in the header.
> > > Something like make
> > > static final int MAX_MINOR_VERSION = 4.
> > > Introduce a NEW_CONSTANT =4 in the HFileBlock
> > > if(minorVersion >= NEW_CONSTANT)
> > > {
> > > //Add new bytes.
> > > }
> > > where minorVersion = MAX_MINOR_VERSION .
> > > So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the
> new
> > > one that i introduce now goes to 4. What i was thinking is increasing
> by
> > 1
> > > everywhere in the code should help but seeing the code that is not the
> > > case.
> > > So is this approach valid or am i missing something? Testing with this
> > > approach am able to make an existing version of Trunk to work with new
> > > changes that i had made.
> > >
> > > Regards
> > > Ram
> > >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Minor versions for HFile and HFileBlock
Posted by ramkrishna vasudevan <ra...@gmail.com>.
The FixedFileTrailer uses MAX_MINOR_VERSION. MINOR_VERSION_WITH_FAKED_KEY
is being checked to use the INDEX_MAGIC but not sure where it is written to
the trailer.
Some of the Minor versions are spread across the code i felt. Will check on
that too.
Regards
Ram
On Thu, Jun 20, 2013 at 5:10 PM, Ted Yu <yu...@gmail.com> wrote:
> Take a look at:
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
>
> static final int MAX_MINOR_VERSION = 3;
>
> /** Minor versions starting with this number have faked index key */
> static final int MINOR_VERSION_WITH_FAKED_KEY = 3;
>
> I think the new constant should be added in HFileReaderV2.java so that it
> is easier to observe all the changes to minor version in one place.
>
> Cheers
>
> On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com> wrote:
>
> > Minor versions are basically for the HFileBlock format. So any change
> that
> > we try to do in the HFileBlock format we need to increase the minor
> > version.
> >
> > The FixedFileTrailer has the minor version 3 written in it.
> > static final int MAX_MINOR_VERSION = 3.
> > This forms the min version that is written to the FixedFileTrailer of the
> > HFile.
> >
> > Inside the HFileBlock constructor
> > {code}
> > if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
> > this.checksumType = b.get();
> > this.bytesPerChecksum = b.getInt();
> > this.onDiskDataSizeWithHeader = b.getInt();
> > } else {
> > this.checksumType = ChecksumType.NULL.getCode();
> > this.bytesPerChecksum = 0;
> > this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
> >
> > HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
> > }
> > {code}
> > where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and minorVersion
> =
> > MAX_MINOR_VERSION.
> >
> > So if i need to make some changes to the HFileBlock format adding some
> > additional bytes to the header, i need to increase the minorversion
> number.
> > But since the minor version is already 3 i need to introduce another
> > constant which is greater than 3
> > and then add the new bytes in the header.
> > Something like make
> > static final int MAX_MINOR_VERSION = 4.
> > Introduce a NEW_CONSTANT =4 in the HFileBlock
> > if(minorVersion >= NEW_CONSTANT)
> > {
> > //Add new bytes.
> > }
> > where minorVersion = MAX_MINOR_VERSION .
> > So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the new
> > one that i introduce now goes to 4. What i was thinking is increasing by
> 1
> > everywhere in the code should help but seeing the code that is not the
> > case.
> > So is this approach valid or am i missing something? Testing with this
> > approach am able to make an existing version of Trunk to work with new
> > changes that i had made.
> >
> > Regards
> > Ram
> >
>
Re: Minor versions for HFile and HFileBlock
Posted by Ted Yu <yu...@gmail.com>.
Take a look at:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
static final int MAX_MINOR_VERSION = 3;
/** Minor versions starting with this number have faked index key */
static final int MINOR_VERSION_WITH_FAKED_KEY = 3;
I think the new constant should be added in HFileReaderV2.java so that it
is easier to observe all the changes to minor version in one place.
Cheers
On Thu, Jun 20, 2013 at 3:16 AM, ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com> wrote:
> Minor versions are basically for the HFileBlock format. So any change that
> we try to do in the HFileBlock format we need to increase the minor
> version.
>
> The FixedFileTrailer has the minor version 3 written in it.
> static final int MAX_MINOR_VERSION = 3.
> This forms the min version that is written to the FixedFileTrailer of the
> HFile.
>
> Inside the HFileBlock constructor
> {code}
> if (minorVersion >= MINOR_VERSION_WITH_CHECKSUM) {
> this.checksumType = b.get();
> this.bytesPerChecksum = b.getInt();
> this.onDiskDataSizeWithHeader = b.getInt();
> } else {
> this.checksumType = ChecksumType.NULL.getCode();
> this.bytesPerChecksum = 0;
> this.onDiskDataSizeWithHeader = onDiskSizeWithoutHeader +
>
> HConstants.HFILEBLOCK_HEADER_SIZE_NO_CHECKSUM;
> }
> {code}
> where static final int MINOR_VERSION_WITH_CHECKSUM = 1 and minorVersion =
> MAX_MINOR_VERSION.
>
> So if i need to make some changes to the HFileBlock format adding some
> additional bytes to the header, i need to increase the minorversion number.
> But since the minor version is already 3 i need to introduce another
> constant which is greater than 3
> and then add the new bytes in the header.
> Something like make
> static final int MAX_MINOR_VERSION = 4.
> Introduce a NEW_CONSTANT =4 in the HFileBlock
> if(minorVersion >= NEW_CONSTANT)
> {
> //Add new bytes.
> }
> where minorVersion = MAX_MINOR_VERSION .
> So my concern is one Constant MINOR_VERSION_WITH_CHECKSUM = 1 and the new
> one that i introduce now goes to 4. What i was thinking is increasing by 1
> everywhere in the code should help but seeing the code that is not the
> case.
> So is this approach valid or am i missing something? Testing with this
> approach am able to make an existing version of Trunk to work with new
> changes that i had made.
>
> Regards
> Ram
>