Overloaded >> operator not reading more than first line from data file

[toneranger]

I'm reading a big csv file into a data structure defined as such:

struct PriceInfo
{
	double       Open;
	double       High;
	double       Low;
	double       Close;
	unsigned int Volume;
	unsigned int Time;
	std::string  Date;
};

So somewhere in my main function, I have this line:

// read contents of datafile by overloading the input operator into a vector of type <PriceInfo> called prices

while (! dataFile.eof())
{ 
   dataFile >> prices 
}

The csv file is comma delineated, so I use an extra function to do the parsing, beyond a template and the overloading function (with the following header std::istream& operator>>(std::istream& istream, PriceInfo& priceInfo))to read the data in.

But the problem is that only the first line in the dataFile is being read into prices.

How should I troubleshoot this?

Thanks,
TR

[Insensus]

Maybe you would also like to post your operator>> because I'm not able to solve it like this.
Perhaps someone with actual coding skills might be able to, though, but to me it seems unlikely.

[toneranger]

Maybe you would also like to post your operator>> because I'm not able to solve it like this.
Perhaps someone with actual coding skills might be able to, though, but to me it seems unlikely.

std::istream& operator>>(std::istream& istream, PriceInfo& priceInfo)
{
	std::string ignored;
	std::string line;
	std::getline(istream, line);
	std::istringstream lineStream(line);
	ReadCommaSeparatedValue(lineStream, ignored); // ignore field 0
	ReadCommaSeparatedValue(lineStream, ignored); // ignore field 1
	ReadCommaSeparatedValue(lineStream, priceInfo.Date);
	ReadCommaSeparatedValue(lineStream, priceInfo.Time);
	ReadCommaSeparatedValue(lineStream, priceInfo.Open);
	ReadCommaSeparatedValue(lineStream, priceInfo.High);
	ReadCommaSeparatedValue(lineStream, priceInfo.Low);
	ReadCommaSeparatedValue(lineStream, priceInfo.Close);
	ReadCommaSeparatedValue(lineStream, priceInfo.Volume);
	// ReadCommaSeparatedValue(lineStream, ignored); // ignore field 9
	return istream;
}

template <typename T>
void ReadCommaSeparatedValue(std::istream& istream, T& t)
{
	std::string value;
	std::getline(istream, value, ',');
	std::istringstream valueStream(value);
	valueStream >> t;
}

template <typename T>
std::istream& operator>>(std::istream& istream, std::vector<T>& vector)
{
	std::copy(std::istream_iterator<T>(istream), std::istream_iterator<T>(), std::back_inserter(vector));
	return istream;
}

[danb737]

To read your file, did you try instead :

while (dataFile >> prices)
{  
}

I think eof() will only return true when you actually tried to read beyond the end of file.

[Insensus]

Ok this will probably not solve it (I'm going to experiment with your code a bit later to find a solution) but there's at least one thing that's kind of odd.

You are looping the action of extracting something from the stream into prices with your overloaded operator>>, but your overloaded operator>> reads the given stream until it reaches the end.
So to my understanding your while loop will only execute once because after the first extraction you should be at the end of the stream.

As for a solution to your original problem, I haven't figured that out yet but maybe you could try skipping the copy algorithm and doing it with loops for now so you can output some more information on what's going on.

[danb737]

I suspect that std::copy is not doing what you think it's doing.

The behavior of this function template is equivalent to:

template<class InputIterator, class OutputIterator>
  OutputIterator copy ( InputIterator first, InputIterator last, OutputIterator result )
{
  while (first!=last) *result++ = *first++;
  return result;
}

So you're taking the first element from the stream, which is probably your ignore field 0 and try to assign it to a PriceInfo object which will be inserted in your vector. But you're not calling the operator>>(istream&, PriceInfo&).

I would write

PriceInfo prInfo;
while(dataFile >> prInfo)
{
    prices.push_back(prInfo);
}

[danb737]

After some thoughts and readings, I think I was wrong, and copy works as you intended.

I would try anyway the push_back idea to make sure that function operator>>(istream&, PriceInfo&) is working as intended. Maybe you could post two lines of your csv file ?

[danb737]

I tried this code (similar to yours)

#include <iostream>
#include <iterator>
#include <vector>
#include <algorithm>
#include <string>
#include <sstream>
#include <fstream>

struct PriceInfo
{
	double       Open;
	double       High;
	double       Low;
	double       Close;
	unsigned int Volume;
	unsigned int Time;
	std::string  Date;
};

template <typename T>
void ReadCommaSeparatedValue(std::istream& istream, T& t)
{
	std::string value;
	std::getline(istream, value, ',');
	std::istringstream valueStream(value);
	valueStream >> t;
}

std::istream& operator>>(std::istream& istream, PriceInfo& priceInfo)
{
	std::string ignored;
	std::string line;
	std::getline(istream, line);
	std::istringstream lineStream(line);
	ReadCommaSeparatedValue(lineStream, ignored); // ignore field 0
	ReadCommaSeparatedValue(lineStream, ignored); // ignore field 1
	ReadCommaSeparatedValue(lineStream, priceInfo.Date);
	ReadCommaSeparatedValue(lineStream, priceInfo.Time);
	ReadCommaSeparatedValue(lineStream, priceInfo.Open);
	ReadCommaSeparatedValue(lineStream, priceInfo.High);
	ReadCommaSeparatedValue(lineStream, priceInfo.Low);
	ReadCommaSeparatedValue(lineStream, priceInfo.Close);
	ReadCommaSeparatedValue(lineStream, priceInfo.Volume);
	// ReadCommaSeparatedValue(lineStream, ignored); // ignore field 9
	return istream;
}

std::ostream& operator<<(std::ostream& ostream, PriceInfo& priceInfo)
{
	ostream << "Date:" << priceInfo.Date << " Time:" << priceInfo.Time
        << " Open:" << priceInfo.Open << " High:" << priceInfo.High
        << " Low:" << priceInfo.Low << " Close:" << priceInfo.Close
        << " Volume:" << priceInfo.Volume << std::endl;
	return ostream;
}

template <typename T>
std::istream& operator>>(std::istream& istream, std::vector<T>& vector)
{
	std::copy(std::istream_iterator<T>(istream), std::istream_iterator<T>(), std::back_inserter(vector));
	return istream;
}

int main()
{
    std::ifstream dataFile("data.txt");
    std::vector<PriceInfo> prices;

    dataFile >> prices;

    std::copy(prices.begin(), prices.end(), std::ostream_iterator<PriceInfo&>(std::cout,""));

    return 0;
}

With these data
data.txt

0,1,13/02/2011,1230,10,20,5,13,500
0,1,14/02/2011,1530,20,30,10,15,1000

And it's working properly... I get two lines with the first two values ignored.
Of course my code is not checking if dataFile is ok. But it was just to find out if it's working or not. Notice I didn't use a while loop but just once the >> operator.

[toneranger]

Akhena,

Sorry for not writing earlier, I was gone over weekend.

Ok, looking at your latest code, it seems like the problem can't possibly be with this line:

dataFile >> prices;

because you got it to work with those two lines, so the operator looks fine.

That can only mean that the rest of my code which uses a sorted set to sort the data over a range of dates contains the problem. Here it is:

smallSet sortedLows;
		for (std::vector<PriceInfo>::const_iterator it = prices.begin(); it!= prices.end(); ++it)
		{
	       std::string lowerDate, upperDate;
		   std::cout << "Please enter lowerDate:" << std::endl;
		   std::cin >> lowerDate;
           std::cout << "Please enter upperDate:" << std::endl;
		   std::cin >> upperDate;

	        {
		        if (it->Date >= lowerDate && it->Date <= upperDate)
			      sortedLows.insert(*it);
	        }

		if (sortedLows.empty())
			std::cout << "sortedLows container is empty." << std::endl;
		else
			std::cout << "sortedLows container is not empty." << std::endl;
    
	PriceInfo lowest = *(sortedLows.begin());
	std::cout << "The lowest price occurred at:" << lowest.Date << " " << lowest.Time;
	

	return EXIT_SUCCESS;
		}

where

struct LowestPrice
{
	bool operator () (const PriceInfo& a, const PriceInfo& b) const
	{
	return a.Low < b.Low;
	}
};


typedef std::set<PriceInfo, LowestPrice> smallSet;

I tried to upload an excel file with some data in it but it won't let me...

[danb737]

I don't know if you did some mistake in your copy-paste, but this code seems to have a few issues...

You're asking for lowerDate and upperDate within the for loop. Is this really intended ? I would assume you ask that once, and loop over the vector to insert the relevant elements in the set.

You are comparing lowerDate and upperDate with PriceInfo.Date. All are defined as strings. I guess you don't want that. You want to compare dates chronologically, right ?

[toneranger]

I don't know if you did some mistake in your copy-paste, but this code seems to have a few issues...

You're asking for lowerDate and upperDate within the for loop. Is this really intended ? I would assume you ask that once, and loop over the vector to insert the relevant elements in the set.

You are comparing lowerDate and upperDate with PriceInfo.Date. All are defined as strings. I guess you don't want that. You want to compare dates chronologically, right ?

Shoot, you're right. Those dates are defined in PriceInfo as strings. But what's confusing me is that the program works for the first day, 12/10/2009. But it doesn't when I ask it to find the low price and time at which the low was made for 12/11/2009, the next day. Let me explain the csv file. It's basically 10 fields, the first 2 are ignored, then it starts with date, time, open, high, low, close, volume, and then 1 last field ignored again. And for one day, the data spans from 9.30am to 16.14pm. So would look like this

- - 12/10/2009 9.30 5, 6, 7, 8, 100
12/10/2009 9.31 7, 6, 8, 9, 120
12/10/2009 9.32 ...
...
12/10/2009 16.14 10, 5, 8, 10, 200

then the next day starts in same way, spanning from 9.30 to 16.14. What I wanted to do was be able to scan the range of each day, and find the low price in that day as well as the time at which it was made. That involves scanning a range of prices over one date, which itself ranges over a time frame of 9.30 to 16.14.

[danb737]

It seems you'll need to be able to compare dates and times. I would give Boost.Date_Time a read. It can parse your strings into meaningful objects that you can compare.