Hello all,
I have what at first glance seems like a very simple problem but I have been getting bogged down with it for the last few days now and would appreciate some help.

I have a series of text files to read in, and wish to convert them to binary files to reduce the file size. The text files contain data arranged in multiple columns and rows - a regular 2 D array. The first row contains text for the column headers, the rest are numbers which could be represented by floats.
Unfortunately the files are space delimited and the number of spaces varies between the entries for each of the columns. Things are further complicated by the fact that the separation is not even consistent between rows either.
My "plan" was to read the file and extract each element and put it into a character array so that I could at least have everything properly organised before sorting out my numbers and converting the whole thing to a binary file.
However I have struggled to get past the first point in this plan. Using some of the previous threads I have initiated my target char arr Table and I have been able to remove the white space from the file, such that the program pulls out each useful bit of text as a char called data , but have lost all the new-line information, and I am not sure how I am going to assign data to an element in my char array. It has been some 5 years since I last did C++ and I am somewhat rusty. My code is below and I have attached an example text file to read in. Any help would be gratefully recieved.

#include <stdlib.h>
#include <stdio.h>
#include <iostream
#include <fstream>
#include <string>
using namespace std;


int main ()
{
	
//------------------------------------------------------------------------------------

//	DECLARATIONS

	typedef unsigned short int us_int;	
   	us_int x;		//an integer counter
	us_int y;		//an integer counter
	const us_int cols=5;	  //the total columns in the array
	const us_int rows=3;  //the rows in the array
	char FileName[80];
	string data;	

	char Table[cols][32]; //char array to hold the data from the file - 31 bits for each element
	char ***TableP;	//pointer for the Table

	TableP = (char ***)malloc(rows * (sizeof *TableP));


//------------------------------------------------------------------------------------


//	STAGE 1: ACQUIRE THE DATA.
//	This stage of the program reads in the text file.

	//	Read the file containing the spaced columns of data.  Each entry is treated as a character (non-numeric)
	//	with the spaces being used to denote separate entities in the array (different numbers)

	//cout << "\nPlease enter the name of your data file, including file extension: ";
	//cin >> FileName;
	cout << "\nThank you\nReading the raw data file...\n";
	//ifstream ReadInFile (FileName);
	ifstream ReadInFile ("s.txt");
	ofstream WriteOutFile ("readout.txt");
	
	x = 0;
	if (ReadInFile.is_open())
	{
		while(ReadInFile >> data)
		{
			x++;
			cout << data << "\t";
			WriteOutFile << data << "\t";
			//cout << x << ": ";
		}
		ReadInFile.close();
		WriteOutFile.close();
	}
	
	else cout << "Unable to open file";
	cout << "\n";
	cout << "\nPress any key and enter to exit!";
	cin >> y;
	cout << "\nDone\n";

//------------------------------------------------------------------------------------


	return 0;
}

just found out I couldn't attach the file so please copy this (from within the inverted commas) and save it as s.txt
" This Is The First Line
This Is The Second Line
1.2343 5.565 1.5485 0.0096 0.76854"

Salem commented: Congratulations on using code tags on your first post. +36
VernonDozier commented: More +rep for code tags on first post. +19
Nick Evan commented: Let's see if you can get 2 blocks for this post :) +22

You can use a structure to contain the data records. ASCII numerical data is converted into binary form and saved as such.
Your ASCII text gets saved into a fixed length buffer within the structue record.

A step below that is you use variable length records. Numerical data is written as individual floats/integers/ or as a structure, Followed by the N bytes of the string. You can either store the length of string before the string, or write a NULL terminator after the ASCII.

There are other tighter ways to pack your data but for purposes of a school assignment, this is sufficient.

Thanks wildgoose,

useful info. I'll look into your tips and let you know how I get on. I noticed somebody posted a similar problem yesterday that I am also looking into. Just to clarify tho I wish I this was for a school assignment, then I wouldn't be so embarrassed by me being over 30 and having to do this for work, although in fairness to myself it has been 5 years since I last did some basic C++ coding, and I have been coding in IDL since.
Cheers

I'm not sure if you have a new question in post#3.
No need to be embarrased. Merely keep in mind a picture an office mate had on their wall about 14 years ago in my I.N.N. days. It was a picture of a dog playing on-line. The message pop-up from the other player, "Wow, you're pretty good!"

In other words. Nobody knows what sex you are, how old you are, etc. The only thing those of us out there like, is clean commented code snippets, that you hopefully single-stepped it in the debugger to attempt to figure out your problem; written in proper written English, and a few other tid bits you'll hear some gripe about from time-to-time.

When the code is too long, or the grammar too bad (I hate texting in emails) I'll flip a mental coin whether to help or not.

WooooHooooooo!!!!!
Just to say I've made a real breakthrough in my problem, using the DaniWeb tutorial on multi dimensional pointers and some other web resources. I have re-thought my approach to the problem, which was to read a text file that contained a 2D array of data, and put each point as an element in array for writing to a binary file.
I use 3D pointers to create a 3D char array ( [rows][cols][data size] ) and use vectors to strip out each line of the text file, and then each non-white space character of the line to get the individual data points. I then use the pointer to assign the data point to the relevant point in my char array. I have yet to convert the chars to real numbers and write this to the binary file, but this should hopefully be a much more trivial matter. If anyone is interested my code is below. I have also attached an example file to read

// Main Code to open text file
// Text file contains 2D data - space delimited and the individual columns are 30-characters long

#include <stdlib.h>
#include <iostream>			
#include <fstream>			
#include <string>
#include <vector>
#include <cstdio>
#include <sstream>

int main()
{

	using namespace std;

//----------------------------------------------------------------------------------------------------------------------------

//	DECLARATIONS

	typedef unsigned short int us_int;		//creating a shorthand for the unsigned short int
	
    us_int i, j, p;							//an integer counter for the loops
	const us_int Rows = 3;					//total number of rows
	const us_int Cols = 31;					//total number of columns
	const int DataSz = 32;					//Number of bytes for data length
	char newline [Cols][DataSz];			//Each line of the full text array -paragraphs
	char ***DataPointer;					//Pointer to each bit of the data - book
	
	char FileName[80];						//Filename to be read
	string line;							//represents a single line from the data file as it is read in


	
	//	The Raw file is converted to a numeric array, RawData, declared on the free store
	//float **RawData;
	//RawData = new float[Rows-1][Cols];

//	Set up the level of detail for the output.  This relates to the number of decimal places shown and recorded
//	The number within the brackets denotes the number of decimal places quoted

	cout.precision(4);

//----------------------------------------------------------------------------------------------------------------------------


//	INITIALISATIONS

	j = p = 0;

	// Allocate the memory for the 3D character array
	DataPointer = (char ***)malloc(Rows * (sizeof *DataPointer));
	if (DataPointer)
	{
		for (i=0;i<Rows;i++)
		{
			DataPointer[i] = (char **)malloc(Cols * (sizeof **DataPointer));
			if (DataPointer[i])
			{
				for (j=0;j<Cols;j++)
				{
					DataPointer[i][j] = (char *)malloc(strlen(newline[j]) + 1 * (sizeof ***DataPointer));
					if (DataPointer[i][j])
						strcpy(DataPointer[i][j],newline[j]);
				}
			}
		}
	}

//----------------------------------------------------------------------------------------------------------------------

//	STAGE 1: ACQUIRE THE DATA.
//	This stage of the program reads in the data

//	The raw input data is in the form of a space delimited text file.  
//	Each line will be read in as a whole an put into a character array for further processing
//	The following is the declaration of the the character array, RawText, into which the raw data is first read into

// Ask user for name of file to be read

	cout << "\n\nPlease enter the name of your data file, including file extension: ";
	cin >> FileName;
	cout << "\nThank you";
	cout << "\nReading the raw data file...";

// Read the file and assign data to RawText

	/*fstream inFile (FileName, ios::in);
	while (! inFile.eof() )
	{
		getline (inFile,line);
		cout << line << endl;
	}
	cout << inFile;
	inFile.close();*/

	vector<string> FileContents;

	ifstream inFile (FileName);
	while(getline(inFile,line ))
	{
		FileContents.push_back(line);
		//cout << "Read Line " << line << endl;
	}

// Each line is now held in the string in FileContents[i]
// Test by printing a few lines	
	cout << "\nFirst Line\n" << FileContents[0];
	cout << "\nSecond Line\n" << FileContents[1];
	cout << "\nThird Line\n" << FileContents[2];


//---------------------------------------------------------------------------------------------------------------------------

// STAGE 2: EXTRACT INDIVIDUAL DATA POINTS
// The data is arranged in 30-character long columns - go along each string/line and
// use stringsrteam to remove all the whitespace and assemble a new vector containing all
// the data points as separate elements.  These can then be assigned to the relevant position 
// in our array using the DataPointer
// remember [row][col][data-length]

	string buffer;								//a buffer string
	string ReadLine;							//a string to hold a particular line from FileContents
	string ReadData;							//a string to hold a particular data point from the line

	for(i=0;i<Rows;i++)
	{
		ReadLine = FileContents[i];
		stringstream ss(ReadLine);					//inserts the FileContents string into a stream

		vector<string> DataPoints;					//a vector, DataPoints, to hold the non-white space characters in the stream

		while (ss >> buffer)
		{
			DataPoints.push_back(buffer);
		}
		
		for(j=0;j<Cols;j++)
		{
			ReadData = DataPoints[j];
			strcpy(DataPointer[i][j], ReadData.c_str())
		}
	}

	cout << "\nThe individual components of the first line are :\n";

	cout << "DP 0,0 " << DataPointer[0][0] << "\n";
	cout << "DP 0,1 " << DataPointer[0][1] << "\n";
	cout << "DP 0,2 " << DataPointer[0][2] << "\n";
	cout << "DP 0,30 " << DataPointer[0][30] << "\n";

	cout << "DP 2,0 " << DataPointer[2][0] << "\n";
	cout << "DP 2,1 " << DataPointer[2][1] << "\n";
	cout << "DP 2,2 " << DataPointer[2][2] << "\n";
	cout << "DP 2,30 " << DataPointer[2][30] << "\n";



//---------------------------------------------------------------------------------------------------------------------------

//	STAGE 3:  CLEANING UP
//	Delete all arrays that were declared on the free store to save memory as they are no longer needed

	//Free the memory
	for (i=0;i<Rows;i++)
	{
		for (j=0;j<Cols;j++)
			free (DataPointer[i][j]);
		free (DataPointer[i]);
	}
	free (DataPointer);

	cout << "Press any key and Enter to exit!\n";
	cin >> j;
   
	return 0;
}

I'll mark this thread as solved when I've got the binary file written.

Forgot attachment!
Wait - can't attach attachment!
You could use this I suppose?

Start Stop number D870 0_sd* error D550 sd* error coefficient coefficient_sd* radius radius_sd* error fraction fraction_sd* d760* d1570* d1660* fraction fraction fraction fraction fraction D550 D670 D865 D760* D1570* D1660* coefficient*
-180.0000 -90.0000 0.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 0.0128 0.0119 0.0106 0.0114 0.0087 0.0085 0.3670
-180.0000 -89.5000 0.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 -999.0000 0.1301 0.1028 0.0729 0.0884 0.0372 0.0348 1.1940

@lost_scotsman - Thanks for posting your solution; was interested how to do it also :)

Hello All,

Just to say I now have my code working exactly as I wished. I use a class to store the text header as a char array and numerical data as a float array, which I can then easily write to a binary file. My code is printed below. Please not that there are a few unncessary comments and screen prints so I could check what was going on but it should give you the general idea. Before I close this thread I would be grateful if people would give it a quick look over and tell me if there is anything I could do to improve the code quality or make it more robust. I was going to put a case in to confirm the file to be opened exists and return a fail if not instead of having the program crash, but if anyone can think of anything else it would be gratefully recieved.

// Main Code to open text file
// Text file contains 2D data - space delimited and the individual columns are 30-characters long

#include <stdlib.h>
#include <iostream>						
#include <string>
#include <vector>
#include <cstdio>
#include <sstream>
#include <fstream>

using namespace std;

//-------------------------------------------------------------------------------------------------------------------------
//  Returns a pointer-to-pointer to a newly created 2D array of size [row,col] 
float **Create2D(int row, int col)
{
	float **Array = new float* [row];
	for (int i=0;i<row;i++)
	{
		Array[i] = new float[col];
	}
	return Array;
}

//  Deletes an array pointed by 'Array' that has 'row' number rows
void Delete2D(float **Array, int row)
{
	for ( int i=0;i<row;i++)
	{
		delete[] Array[i];
	}
	delete [] Array;
}
//  Declare a class 'CompleteData' to hold the character array for the file header and the float array for the numerical data
class CompleteData
{
public:
	char myHeader[31][32];
	float myData[2][31];
};




//-------------------------------------------------------------------------------------------------------------------------
//  BEGIN MAIN PROGRAM

int main()
{

//	DECLARATIONS

	typedef unsigned short int us_int;		//creating a shorthand for the unsigned short int
	
    us_int i, j, p;							//an integer counter for the loops
	const us_int rows = 3;					//total number of rows 259201
	const us_int cols = 31;					//total number of columns
	
	char FileName[256];						//Filename to be read
	char SaveFile[256];						//File data will be saved to
	string line;							//represents a single line from the data file as it is read in

//	Set up the level of detail for the output.  This relates to the number of decimal places shown and recorded
//	The number within the brackets denotes the number of decimal places quoted

	cout.precision(6);

//----------------------------------------------------------------------------------------------------------------------------


//	INITIALISATIONS

	j = p = 0;

	// Declare and initialise the float array to hold the datapoints with the header removed 
	float **Data = Create2D(rows-1,cols);
	

//----------------------------------------------------------------------------------------------------------------------

//	STAGE 1: ACQUIRE THE DATA.
//	This stage of the program reads in the data

//	The raw input data is in the form of a space delimited text file.  
//	Each line will be read in as a whole and put into a vector string 

// Ask user for name of file to be read

	cout << "\n\nPlease enter the name of your data file, including file extension: ";
	cin >> FileName;
	cout << "\nThank you";
	cout << "\nReading the raw data file...";

// Read the file and assign the lines to FileContents
	vector<string> FileContents;

	ifstream inFile (FileName);
	while(getline(inFile,line ))
	{
		FileContents.push_back(line);
		//cout << "Read Line " << line << endl;
	}

// Each line is now held in the string in FileContents[i]
// Test by printing a few lines	
	cout << "\nFirst Line\n" << FileContents[0];
	cout << "\nSecond Line\n" << FileContents[1];
	cout << "\nThird Line\n" << FileContents[2];


//---------------------------------------------------------------------------------------------------------------------------

// STAGE 2: EXTRACT INDIVIDUAL DATA POINTS AND CONVERT TO CHARS AND FLOATS AS APPROPRIATE
// The data is arranged in 30-character long columns - go along each string/line and
// use stringsrteam to remove all the whitespace and assemble a new vector containing all the data points as 
// separate string elements.  The strings are converted to chars with .c_str().
// The first line is kept made into a 2D char array to store the header information.
// The remaining lines are then converted to float numbers using the strtod function and assigned to the 
// relevant position in the target array using the Data pointers

	string buffer;								//a buffer string
	string ReadLine;							//a string to hold a particular line from FileContents
	string ReadData;							//a string to hold a particular data point from the line
	char Header[cols][32];						//a char array to hold the header data

// Extract the header information
	ReadLine = FileContents[0];
	stringstream ss(ReadLine);					//inserts the FileContents string into a stream

	vector<string> DataPoints;					//a vector, DataPoints, to hold the non-white space characters in the stream

	while (ss >> buffer)
	{
		DataPoints.push_back(buffer);
	}
		
	for(j=0;j<cols;j++)
	{
		ReadData = DataPoints[j];
		strcpy(Header[j], ReadData.c_str());
	}

// Extract the data points
	for(i=1;i<rows;i++)
	{
		ReadLine = FileContents[i];
		stringstream ss(ReadLine);					//inserts the FileContents string into a stream

		vector<string> DataPoints;					//a vector, DataPoints, to hold the non-white space characters in the stream

		while (ss >> buffer)
		{
			DataPoints.push_back(buffer);
		}
		
		for(j=0;j<cols;j++)
		{
			ReadData = DataPoints[j];
			Data[i-1][j]=strtod(ReadData.c_str(),NULL);
		}
	}

	cout << "\nThe data from the file is as follows\n";
	for(j=0;j<cols;j++)
	{
		cout << Header[j] << "\t\t";
	}
	for(i=0;i<rows-1;i++)
	{
		cout << "\n";
		for(j=0;j<cols;j++)
		{
			cout << Data[i][j] << "\t\t";
		}
	}
	cout << "\n\n";

//---------------------------------------------------------------------------------------------------------------------------
//	STAGE 3: ASSIGN THE HEADER AND DATA ARRAYS TO THE CLASS 'CompleteData'

	CompleteData DataSet;						//Make DataSet a type of CompleteData
	for (j=0;j<cols;j++)						//Assign the header to the DataSet
	{
		strcpy(DataSet.myHeader[j], Header[j]);
	}
	for(i=0;i<rows-1;i++)						//Assign the numerical data to the DataSet
	{
		for(j=0;j<cols;j++)
		{
			DataSet.myData[i][j] = Data[i][j];
		}
	}

	cout << "\nQuick check - data line 1 30th column \t" << DataSet.myData[0][29] << "\n\n";

	
//---------------------------------------------------------------------------------------------------------------------------

//  STAGE 4: SAVE THE NUMERICAL DATA TO BINARY FILE
	cout << "\nWriting data to file\n";
	cout << "Please chose a file name to save the data to, including file extension: \n";
	cin >> SaveFile;

	ofstream myFile (SaveFile, ios::binary);
    myFile.seekp (0);
 	myFile.write (reinterpret_cast<char *>(&DataSet), sizeof (CompleteData));
	myFile.close();

// Check write is ok
// Set up a new type of CompleteData to hold the information read in from the binary file
	CompleteData NewSet;
// Read in the Binary file and read some of the elements
	ifstream BinaryFile(SaveFile, ios::binary);
	BinaryFile.seekg(0);
	BinaryFile.read(reinterpret_cast<char *>(&NewSet),sizeof(CompleteData));
	BinaryFile.close();

	cout << "Quick check - Header : \t" << NewSet.myHeader[0] << "\t" << NewSet.myHeader[1] << "\n";
	cout << "Quick check - data line 1 30th column \t" << NewSet.myData[0][29] << "\n\n";
		
	BinaryFile.close();



//---------------------------------------------------------------------------------------------------------------------------

//	STAGE 5:  CLEANING UP
//	Delete all arrays that were declared on the free store to save memory as they are no longer needed


	cout << "Press any key and Enter to exit!\n";
	cin >> j;

	Delete2D(Data,rows-1);
   
	return 0;
}

Cheers for all your help and kind comments, and the tutorials on the site have been very useful - new posters check 'em out.:)

cols is 0...cols-1 [0,cols)
rows is both 0... rows-1 [0,rows)
AND
1...rows-2 (0,rows-1)

I don't see the reason for this!

NEW HELP NEEDED!!!!!!
Turns out my joy was short lived. I have been running the programme with some dummy data - 3 lines text files of data similar to what I need to process. however, the real data is some 260,000 lines of data, and as a result I obviously get a stack error due to my class definition. It was originally this:

class CompleteData
{
public:
	char myHeader[31][32];
	float myData[2][31];
};

but changing it to suit my data it would become this:

class CompleteData
{
public:
	char myHeader[31][32];
	float myData[259200][31];
};

So my big question is can I dynamically allocate memory for my class on the free store, and if so how, or do I dynamically set the memory for myData , and if so how?
Any help will be gratefully recieved.

Also to answer WildGoose's thread (if I understood it cotrrectly) the text file contains 'x' rows, but this includes a header at the top of the file. I separate out the header and when I am working only with the numbers the number of rows I have is x-1, hence some of the loops and initialisations have Rows-1.

Sorry, managed to somehow post my response twice...
:$

But some rows were dealt with as

Ex: If 10

0...8
1....9
You were using both!

Assuming you have the memory, you can allocate it with new, instead of having it static! Getting that big you might want to start think of SIMD organizing your data.

Hi WildGoose,

hmmm, the code seems to work ok where the rows are concerned but I'll give it the thorough once over to make sure it is ok.

Thanks for the advice. I think I have the memory to handle this operation (luckily for me it will not be a regular operation) so I may put SIMD on the back burner for now. When you say declare it as new, are you referring to the array within the class or the class itself?

Cheers

OK, I'm about at the end of my tether with this. I have been able to dynamically allocate the memory for my array in my class definition:

class CompleteData
{
public:
      char myHeader[31][32];
      float **myData;
};

and within the main part of my code I use:

CompleteData DataSet;   //Make DataSet a type of CompleteData
//    Step 1: Allocate storage for a 1D array of pointers
//    Step 2: For each pointer, allocate storage for an array of, in this case, floats

      DataSet.myData = new float* [rows-1];    
      for (int i=0;i<rows-1;i++)                           
      {                                                                
            DataSet.myData[i] = new float[cols];  
      }

I feed in my header char array and float array of data to this class as so:

//Assign the header to the DataSet
      for (j=0;j<cols;j++)                                  
      {
            strcpy(DataSet.myHeader[j], Header[j]);
      }
//Assign the numerical data to the DataSet
      for(i=0;i<rows-1;i++)                                 
      {
            for(j=0;j<cols;j++)
            {
                  DataSet.myData[i][j] = Data[i][j];
            }
      }

Even with my massive data arrays of some 260,000 rows this code works - I can get the program to print any of the data points I like to screen using the definition DataSet.myData So far so good. I then go to save my class DataSet as a binary file.

cin >> SaveFile;
      ofstream writeFile (SaveFile, ios::binary);
      if (writeFile.is_open())
      {
            writeFile.seekp (0);
            writeFile.write (reinterpret_cast<char *>(&DataSet), sizeof (CompleteData));
            writeFile.close();
      }
      else
      {
            cout << "Unable to open file - closing down!";
            cout << "\nPress any key and enter to continue!\n";
            cin >> p;
            return 1;
      }

This is where the problem arises. The resultant binary file is very small less than 1kB in size which immediately raises suspicions. On trying to read the file I find that the header information is there but the numbers are not. I check this in the same program by first freeing the memory for my huge array

for (i=0;i<rows-1;i++)
      {                                 
            delete[] DataSet.myData[i];
      }
      delete [] DataSet.myData;

and then try to read in my newly made binary file, setting up a new type of class CompleteData and allocating memory for the array:

// Set up a new type of CompleteData to hold the information read in from the binary file
      CompleteData NewSet;
      NewSet.myData = new float* [rows-1];            
      for (int i=0;i<rows-1;i++)                            
      {                                                                 
            NewSet.myData[i] = new float[cols];       
      }                                                                 
// Read in the Binary file and read some of the elements
      ifstream BinaryFile(SaveFile, ios::binary);
      if (BinaryFile.is_open())
      {
            BinaryFile.seekg(0);
            BinaryFile.read(reinterpret_cast<char *>(&NewSet),sizeof(CompleteData));
            BinaryFile.close();
      }
      else
      {
            cout << "Unable to open file - closing down!";
            cout << "\nPress any key and enter to continue!\n";
            cin >> p;
            return 1;
      }

Using NewSet.myHeader I can access all the header information, but using NewSet.myData to access any of the data points I always get the same value of 4....e+008 (sorry I can't remember the full details of the number). I am guessing I am either making a small but critical error or my whole method is completely wrong. In any case I would gratefully accept any help to explain why my write process obvioulsy isn't working, or whether I am not setting things up correctly to recieve the data.
Thanks in advance

Hi,
just to say that the number that gets printed out for all values of NewData.myData is
-4.31602e+008
I have used this program to open and process a much smaller file (3 lines instead of 260,000) and get the same result. This is really doing my head in so please help me if you can :confused:

lost_scotsman,
996 bytes will be the size of binary file. Calculate the size of an instance manually and put it with write & read methods.

Hi,

I actually just realised that for myself and was about to post an update when I saw your post. I used sizeof() to tell me the size of my class and the elements, such as myData within my class, and lo and behold it is only 4 bytes. That is because my myData is a pointer to a float, and is not the whole array in itself

class CompleteData
{
public:
      char myHeader[31][32];
      float **myData;
};

Gaaaah!!!
So my question is, if I assign the correct size memory for my class in both the read and write commands, will it write and then read all my data?
I'll try it now but any help would also be appreciated!!!

There is no doubt about it. It has to work.

Unfortunately my worst fears were realised. It doesn't matter how large I set the size in the write and read commands, the fact remains that my classification of DataSet.myData in my class CompleteData is just the pointer, and therefore is a single value, so when I tell the program to write DataSet to file it doesn't write the whole array.
Given that I need a large array to hold all my data, which must be allocated on the free store, I find myself asking these questions:

1) Is there a way to declare the whole array on the free store in my class?
2) If not is there a way to write all of my data to the binary file, afterall I can use cout << DataSet.myData[1][30]; to print out the value on the 2nd row and 31st coumn of my data array, so I know that numbers are being attributed to array pointed to by **myData ?

Please help, my mental state is becoming more and more unhinged!:confused:

lost_scotsman,

..in my class CompleteData is just the pointer, and therefore is a single value, so when I tell the program to write DataSet to file it doesn't write the whole array.

Misconceptions about pointer.
Read or write method needs an address of data where it begins to read or write along with total numbers of bytes to be read or written respectively.

I have two code snippets to write some data into binary mode data file.

int totele=20000;
   double *d=new double[totele];
   int i;
   for(i=0;i<totele;i++) {
      d[i]=i;
   }
   int size=totele * sizeof(double);
   fstream f;
   f.open("text.txt",ios::out | ios::binary);
   f.write((char*)d,size);
   f.close();
   delete []d;

and read from the binary mode data file.

int totele=20000;
   double *d=new double[totele];
   int i;

   int size=totele * sizeof(double);
   fstream f;
   f.open("text.txt",ios::in | ios::binary);
   f.read((char*)d,size);
   f.close();
   for(i=0;i<totele;i++) {
      cout << d[i] << " ";
   }
   delete []d;

A big thank you to ADataPost. I understand the read and write commands better now. Unfortunately the code doesn't quite work properly yet.

I used your posted code as an example and tried to apply it to mine

ul_int size = sizeof(CompleteData) + (rows*cols*sizeof(float));
fstream writeFile;
writeFile.open (SaveFile, ios::out | ios::binary);
writeFile.seekp (0);
writeFile.write((char*)&DataSet,size);
writeFile.close();

where CompleteData is my structure to hold my header array and the pointer to the 2D numerical data array

struct CompleteData
{
	char myHeader[31][32];
	float **myData;
};

Because DataSet is a type of CompleteData structure that was not referenced by a pointer I have had to include the unary operator & to give the address, i.e. writeFile.write((char*)&DataSet,size); not writeFile.write((char*)DataSet,size); . The code compiles ok but I am still getting the same result, namely reading from the binary file gives me myHeader but nothing for myData

//Initialise array to hold numerical data pointed to by **myData
CompleteData NewSet;
NewSet.myData = new float* [rows-1];		
for (int i=0;i<rows-1;i++)	
{
NewSet.myData[i] = new float[cols];		
}
fstream BinaryFile;
BinaryFile.open(SaveFile, ios::in | ios::binary);
BinaryFile.seekg(0);
BinaryFile.read((char*)&NewSet,size);
BinaryFile.close();

Also when the program is finished I get the following error Unhandled exception at 0x104d438c (msvcp80d.dll) in NewFileReadBinaryOut_V5.exe: 0xC0000005: Access violation reading location 0x00000004. This points to fstream and the following

virtual __CLR_OR_THIS_CALL ~basic_fstream()
		{	// destroy the object
		}

I don't know quite what is causing that problem.
The good news is that if I follow your code but only write the numerical data I had stored in an array before writing it to the structure I get a proper read and write operation. This will do for now but I would like the header information to be included so if anybody could help resolve this last little issue I would be very grateful.
Thanks everyone!

Your problem is that you are still writing pointers into a file. Write only the data not any address.
Write:

char ar[3][10];
   double **p;

   strcpy(ar[0],"No1");
   strcpy(ar[1],"No2");
   strcpy(ar[2],"No3");
   int rows=100,cols=3;
   int i;

   p=new double*[rows];
   for(i=0;i<rows;i++)
    {
       p[i]=new double[cols];
       for(int j=0;j<cols;j++)
        {
          p[i][j]=j;
        }

    }
    int size=cols*sizeof(double);
   fstream f;
   cout << "\n" << size;

   f.open("t.txt",ios::out | ios::binary);
   f.write((char*)ar,sizeof(ar));
   for(i=0;i<rows;i++)
     f.write((char *)p[i],size);
   f.close();

   for(i=0;i<rows;i++)
    {
      delete []p[i];
     }
    delete []p;

Read:

char ar[3][10];
   double **p;

   int rows=100,cols=3;
   int i;

   p=new double*[rows];
   for(i=0;i<rows;i++)
    {
       p[i]=new double[cols];
    }
    int size=cols*sizeof(double);
   fstream f;
   cout << "\n" << size;
   f.open("t.txt",ios::in | ios::binary);
   f.read((char *)ar,sizeof(ar));
   cout << "\n" << ar[0] << "  " << ar[1] << "  " << ar[2] << "\n";
   for(i=0;i<rows;i++)
    {
     f.read((char*)p[i],size);
    }
   f.close();
   for(i=0;i<rows;i++)
    {
       for(int j=0;j<cols;j++)
        {
          cout << "  " << p[i][j];
        }
        cout << "\n";
    }
   for(i=0;i<rows;i++)
    {
      delete []p[i];
     }
    delete []p;
commented: Succinctly helped me with my problem - understood what I needed and made it clear what I had to do - Excellent advice!!! +1

Hi ADataPost,
You have been so helpful. I didn't know that I could read and write within a for loop, nor did I know that it was possible to write multiple data types (char, doubles etc) to file and pull them out again. I have run you code and I understand it. My code has been turned over so many times that I am going to re-write it and implement your read/write functions. I am confident that this should solve the problem and will get back to you as soon as I can confirm this. Thanks for you help once again - you've been a real help!!!

A final thank you to adatapost. The problem is solved, the program works beautifully and the results are staggering.
The text files were 230MB in size, as a binary they are 30MB. To read and pull data from the text file took 2 minutes, to pull the same data from the binary file took little over 2 seconds! Thanks to all the posters who have helped and to those who left some nice comments.
Cheers!

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.