Bitwise Encoding

[llemes4011]

Hi. I have some data that I need to store in a file, and (for simplicity's sake) I was considering using bitwise operators to pack it all into a single hex or dec number. I've never user bitwise operators, as they really confuse me. If anyone has any pointers and links to pages with good explanations, or knows of a better way to store multiple ints in a way that is easily retrievable, please let me know.

Thanks!!!

Also. I wanted it to be stored as so:

0xXXXXXXXX I know that's probably to big to store as an int, but each pair (XX) need to store data for an int (Yes, the int's size would be less than 256). but now I think im rambling because I have no idea how this kind of thing works! hahaha

[wildgoose]

Bit Magic. You need to post the types of data you wish to store and what the maximum number range is for each!

There are many books on the subject! Did you know you can use bit data instead of code branching for faster code in some cases?

But keep in mind Java is very restricting unlike C in this regard!

[llemes4011]

int: 0-6, double: 0-30 (But I can change it to int if need be), double: 0-30, int: 0-?? (Codes for images, probably end up being about 100ish)

huh. this stuff sounds really neat, I wish it was less confusing >.<

[wildgoose]

0-6 = XXX 3 digits

double (DPFP) Double-Precision Floating-Point ???
Do you really need a double?
They're 64 bits each SPFP are 32-bits.
But if you can store in fixed point you can get it down. What's the whole number versus how many precision digits?

Image codes, enum's? How many images maximum?

Once you figure this out, then its a matter of bit masking and shifting!

[llemes4011]

The doubles should probably be floats in my program, i learned decimals in programming on doubles, so that's what I've used.

For image code, I meant that the image stored referrs to a specific Image elsewhere(I'll just compare the number to in index of the Image in the list of Images... or something like that. ) but there wouldn't be any more than 100, and absolute maximum.

The doubles only go out 1 or 2 places significantly (yeah, I should be using floats >.<).

[wildgoose]

2^N so 100 is 7 bits 0...127 XXX XXXX

You can pack your floating-point!

For example if your number ranges are 0.0 to 31.0 then the whole number can be stored in 5 digits in integer from 0...31 XXXXX
And assuming a resolutions of 0.001 is sufficient then that's 1/1000 So 1024 10 digits. So 5+10 15 bits to store your 64-bit value!

If you're talking angles 0 to 360 degrees, essentially 2 PI.
Then 0 to 4095. 12 bits to store a 360 degree angle.

[llemes4011]

I am so confused O.o I don't know what any of that means >.< Are you saying that I can store all of my data in a 15 digit long number?

[wildgoose]

You need to analyze one record of data for maximum bit requirements each field.
And need to know exactly what data ranges are stored in the floating-point.

[llemes4011]

Okay, so I know this:

field1(int)   range = 0 to 5
field2(float) range = 0.00 to 30.00 (need 2 sig-figs)
field3(float) range = 0.00 to 30.00 (need 2 sig-figs)
field4(int)   range = 0 to 100

And with that knowledge I can pack each object's data that needs to be saved into less than 20 bits of data per object??? How is that done (You've probably already answered this, but I took it as something else if you did...)

[JamesCherrill]

Min bitsizes:
0-5 : 4 bits
0.00 - 30.00 : 12 bits
0-100 : 7 bits
so 20 isn't enough, theoretical min is 35 bits
Sensible compromise may be to store the int values in byte variables, and the floats as short variables (held in units of 1/100 - eg 1.23 stored as 123) for a total of 48 bits and no need to do any bit manipulation at all.
ps I hope you need to store millions of these, because otherwize it's all a waste of time. A few bytes here or there don't matter today.

[~s.o.s~]

> 0-5 : 4 bits
I wonder how you came up with 4. ;-)

@llemes4011
Coming up with your own serialization mechanism is hard; continue if you are doing this for fun but do remember that there are many other good binary serialization formats out there which can lessen your work and reduce the agony. A comparison of various serialization libraries can be found here.

[JamesCherrill]

> 0-5 : 4 bits
I wonder how you came up with 4. ;-)

Oh yes, thanks. I jumped straight to ( smallest n where 2^n >= 5 ), but it should be (2^(n+1) - 1) >= 5
Sorry OP, subtract 1 bit from my previous post.
J.

[llemes4011]

ps I hope you need to store millions of these, because otherwize it's all a waste of time. A few bytes here or there don't matter today.

closer to around 40,000 >.< I was hoping that it would be a bit easier and more convenient than this for my needs >.<

[JamesCherrill]

Well, I would definitely say that for 40,000 it would be worth using byte and short (as per my previous post), but not worth trying to pack the bits any tighter.

[~s.o.s~]

> closer to around 40,000

40,000 is nothing, unless you have some specifications which you would need to meet and you have *confirmed* that doing things your way is much faster than the other simpler approaches.

IMO just go with the normal way of doing things i.e. writing data as comma separated values. If you are feeling adventurous, rather screw around with an embeddable database than get involved with all the bit-fiddling. Optimize when needed and that too after profiling; premature optimization is the root of all evil.

[wildgoose]

field1(int) range = 0 to 5 {3}
field2(float) range = 0.00 to 30.00 (need 2 sig-figs) {5+7=12}
field3(float) range = 0.00 to 30.00 (need 2 sig-figs) {5+7=12}
field4(int) range = 0 to 100 {3}

So 30 bits total. 32-bit integer so 2 spare bits. What fields should those two bits be given to for future expansion?

so 32/192 compaction ratio.

[JamesCherrill]

Hey! Now it's my turn!

> 0-100 : 3 bits
I wonder how you came up with 3. ;-)

Anyway I agree with sos. Unless you need to store this data on punched cards, the odd few 10k bytes here or there is pretty much irrelevant.

[llemes4011]

Ok, thanks for all the help you guys, this stuff is way over my head anyways =P Maybe I'll understand it some day, lol, for now im going to stick with just writing the values to the file.