Writing to file - corrupted data?

[martin_anastaso]

Hello all,

I have encountered a very bizzare (at least for me) problem when writing the output from my program to a file. I've written a program to solve numerically mean-reverting SDEs. The program takes some parameters and returns several dynamic arrays corresponding to the number of SDEs I have. I store each realization in a row; the columns correspond to the discretized time steps.

Seems to be working fine but sometimes when I write the results to a .txt or .csv file, some of the data elements are "-1.#IND". Doesn't always happen and seems to be correlated with the size of the arrays I'm working with. With smaller arrays there is usually no problem. When the size increases I sometimes get this kind of corrupted data but not always.

The problem is the whole thing is unstable and I can never be sure if I'll be able to replicate my results. I searched on the Internet for this but did not find anything. Does anyone have an idea what -1.#IND means, in the first place, and what might be potentially causing this problem?

I can think if two possible explanations. One is that this is due to some numerical error, but this is unlikely since the values prior to the -1.#IND elements are perfectly fine, then I suddenly get some sequence of -1.#IND elements and then normal values again. The second reason I can think of is that something happens during the writing of the data to the file. Again, this shouldn't be the case as the size of the arrays I'm storing does not exceed the file's storage potential.

I'm lost. Hope someone can help me resolve this. Thank you very much in advance.

Martin

[Unimportant]

You didn't post any code snippets, but it seems like you're reading or inputting uninitialized data.

[martin_anastaso]

OK, my bad. Here is a short snippet of the actual computational routine. Note that within the loop divs[k][j+1] is not undefined since I first fill divs by calling another function which gives me the mean reverting values. So within the loop, for each successive step it's the mean reverting value minus the value during the previous period. I've stepped through the code - the problem is if an error occurs it happens when I have many rows (with many columns for each) in each array and this means I could be stuck stepping through the code all day if I want to find exactly what's causing the problem. And there is no guarantee an error will occur. Is there some way to set a conditional breakpoint to pause the execution once this -1.#IND value occurs? Thanks.

for (std::vector<dynamicArray<divType>>::size_type k = 0; k < divs.size(); ++k)		//For each asset...
	{
		for (std::vector<divType>::size_type i = 0; i < divs[k].size(); ++i)		
		{
			divs[k][i].insert(divs[k][i].begin(),DtZero[k]);							//Add the starting dividends as a first column.

			for (std::vector<divType>::size_type j = 0; j < (divs[k][i].size()-1); ++j)
			{
				dW = sqrt(dt) * std_norm_Box_Muller();

				if (CIRDummy == 1)
				{
					//Compute solution using CIR process.
					divs[k][i][j+1] = divs[k][i][j] + 
						((alpha * (divs[k][i][j+1] - divs[k][i][j])) * dt) + 
						((beta * sqrt(divs[k][i][j])) * dW) + 
						((0.25 * (pow(beta,2))) * ((pow(dW,2)) - dt));
				}

				if (CIRDummy == 2)
				{
					//GOU. Reflection principle.
					divs[k][i][j+1] = divs[k][i][j] + 
						(((alpha * (divs[k][i][j+1] - divs[k][i][j])) * divs[k][i][j]) * dt) + 
						((beta * divs[k][i][j]) * dW) + 
						((0.5 * (pow(beta,2)) * divs[k][i][j]) * ((pow(dW,2)) - dt));
					divs[k][i][j+1] = abs(divs[k][i][j+1]);
				}
			}
		}
	}

[Unimportant]

-1.#IND -> undefined (NaN)
-1.#INF -> infinity

I see you are doing quite a bit of math with potentially complex numbers, perhaps sqrt sometimes gives you a NaN which may be the source of your problem.

Such as -> ((beta*sqrt(divs[k][j])) * dW) + // where dW may be NaN, making the entire value NaN.

What I mean to say is that this is not corrupted data, but a mathematical problem.

Sorry I do not know what type of math you are really doing and cannot provide insight into which part may be giving you "NaN" values. The above is just a guess/example to help point you in the right direction.

Hope that helps!

[martin_anastaso]

Thank you for your time, Unimportant :)

[martin_anastaso]

OK, so I have finally localized the problem. For some unexplainable reason the expression

dW = sqrt(dt) * std_norm_Box_Muller();

sometimes evaluates to either 1.#INF or -1.#INF. I don't understand how this is possible. The function call std_norm_Box_Muller() gives a standard normal random variable (mean = 0, standard deviation = 1). dt is usually 0.01. How can this ever produce + or - infinity? I tested just this expression separately and in one million trials I did not get even one such value so it is not the case that std_norm_Box_Muller() gives me some very large value. I'm truly lost...

[Unimportant]

Is dt ever negative?
That could give you an undefined value depending on what complex math classes you are using.

In the real number system, a negative number has no square root (imaginary numbers required), as any number multiplied by itself will be positive.

As a quick test, why don't you try this:
sqrt(abs(dt))

and see if the NaN outputs are removed, if so, as this would lead to incorrect answers, you will likely have to create or use a class which can support complex number mathematics (such as imaginary numbers).

Best of luck!

[martin_anastaso]

Nahh, dt is never negative. Besides, I started storing everything in additional arrays to see what's happening - the culprit definitely is the call to std_norm_Box_Muller(). Inexplicably sometimes it just returns infinite values. It is not the actual function, though, since I tried it separately by generating random numbers and no infinite values occur there. This is exactly what bugs me - everything works like clockwork separately but when I start calling std_norm_Box_Muller() as in the code above, sometimes this problem occurs. If only I could figure out why...

I've noticed that it is usually only one such infinite value in the sequence of random numbers that messes everything up so there is a solution - make a do-while loop to generate additional random numbers if infinity is ever encountered but this is somewhat artificial and does not solve the real problem...