Efficient way to write and read huge boost dynamic_bitset to & from a file

[SyncMr]

0 down vote favorite

I have a huge vector of boost dynamic_bitset. I want to write the dynamic_bitset vector to a file and later read the file back into a dynamic_bitset vector. Is the memory for dynamic_bitset allocated as a contiguous block of memory (so that I can write the entire vector at once without traversing) ?

The size of the bitset vector is in order of millions. So I am looking for an efficient way to write them to a file instead of iterating through the elements.

I converted the dynamic_bitset to a string and then wrote the string to a file. Later read the string from the file and converted it back to dynamic_bitset.

Below is the code I wrote in C++ using Visual Studio:

#include "stdafx.h"
#include <iostream>
#include <fstream>
#include <string>
#include <boost/dynamic_bitset.hpp>
using namespace std;

int main(int argc, char* argv[])
{
  // Initializing a bitset vector to 0s
  boost::dynamic_bitset<> bit_vector(10000000); 
  bit_vector[0] = 1;   bit_vector[1] = 1;  bit_vector[4] = 1;  bit_vector[7] = 1;  bit_vector[9] = 1;
  cout<<"Input :"<<bit_vector<<endl; //Prints in reverse order

  //Converting dynamic_bitset to a string
  string buffer;
  to_string(bit_vector, buffer);

  //Writing the string to a file
  ofstream out("file", ios::out | ios::binary);  
  char *c_buffer = (char*)buffer.c_str();   
  out.write(c_buffer, strlen(c_buffer));
  out.close();

  //Find length of the string and reading from the file
  int len = strlen(c_buffer);
  char* c_bit_vector = new char(len+1);
  ifstream in;
  in.open("file", ios::binary);
  in.read(c_bit_vector, len);
  c_bit_vector[len] = 0;
  in.close();

  //Converting string back to dynamic_bitset
  string str2 = c_bit_vector;
  boost::dynamic_bitset<> output_bit_vector( str2 );
  cout<<"Output:"<<output_bit_vector<<endl;

  system("PAUSE");
  return 0;
}

But even this method, storing it as a string, takes a long time to write to the file. And when I try to read back from the file into the string, I get an "unhandled access violation exception". Is there a more efficient way to implement the same? Thanks for your help

[NathanOliver]

according to this http://www.boost.org/doc/libs/1_36_0/libs/dynamic_bitset/dynamic_bitset.html
You ahould be able to output direclty to your filestream using the << operator. They also have the >> operator set to get a bitset from the stream.

[SyncMr]

@NathanOliver:
1) But for writing a bitvector of size(1000000000) using the filestream operator takes 1min 30 seconds.
2) It does not store values as bits instead it stores them as bytes. i.e. a bitset of size 100 occupies 100 bytes.
3) Using << operator to write a dynamic_bitset to a file is same as traversing the bitset element by element and using ofstream to write to a file. Internally both follows the same implementation.

I am looking for some method
1) Using which, I can write the actual bits instead of bytes so the file size would be less.
2) In which I can store elements as contiguous block of memory and write them at once to the file and later read it back at once.

[NathanOliver]

Do you really need 1000000000 bits in one container? What exactly are you trying to do? The problem with bits is that they can’t be stored singularly. The smallest type is char/bool which is one byte. You could do operations to concatenate the bits into some data type and then write that to the file to save space but it can get very complicated.