I have a CSV that i can parse no problem at 10am..
But by 5pm it does grow to over 100,000 lines that cuases the script to hang.
I have to parse the CSV into an array that i can then shape.
If anyone has any idea how i can parse this without the timeouts, or a better solution tahn fgetcsv that would be appreciated.
thank you,
Below is where it usually starts to hang around. I'm sure i'm just doing this very inefficiently, and there probably is a better solution.
require_once('../../Connections/dom/simple_html_dom.php');
#url : http://mis.nyiso.com/public/csv/realtime/20111212realtime_gen.csv
$url = "http://mis.nyiso.com/public/csv/realtime/"; // The simple url
$name = "realtime_gen.csv"; // The parameter name
$date = date("Ymd");
$newUrl = $url . "$date$name"; // appending the values
echo $newUrl;
// begin indexed array
$data=array();
$fp = fopen($newUrl, "r");
$names = array_flip(fgetcsv($fp, 5000000));
while (($values = fgetcsv($fp, 5000000)) !== FALSE) {
# print "The ".$values[$names["Name"]]." says ".$values[$names["Time Stamp"]].".\n";
$data[]=$values;
}
fclose($fp);
# ----------- Explode time/date to proper format
$explodeArr = array();
$theData = array();
$c = count($data);
for($i = 0; $i < $c; $i++)
{
$explodeArr[] = explode(" ",$data[$i]['0']);
unset($data[$i]['0']);
$mergeArray = array_merge($explodeArr[$i],$data[$i]);unset($data[$i]);
$theData[] = $mergeArray;
}unset($data);
# ->>>>>>>>>>>>>>>>>>>>>>>>>>SECTION OFF GENERATORS TO OWN ARRAY
$nyc = array();
$millwood = array();
$hudvl = array();
$dunwood = array();
$c=count($theData);foreach ($theData as $i=>$v){
if (preg_match("/(23512|23513|23515|23516|23517|23518|23519|23520|23523|23524|23533|23534|23535|23538|23540|23541|23657|23660|23729|23770|23786|23810| 23816|23817|23820|24077|24078|24079|24080|24084|24094|24095|24096|24097|24098|24099|24100|24101|24102|24103|24104|24105|24106|24107| 24108|24110|24111|24112|24113|24114|24115|24116|24117|24118|24119|24120|24121|24122|24123|24124|24125|24126|24127|24128|24129|24130| 24131|24132|24133|24134|24135|24136|24137|24138|24149|24152|24155|24156|24157|24158|24159|24160|24161|24162|24163|24168|24195|24196| 24202|24225|24226|24227|24228|24229|24230|24231|24232|24233|24234|24235|24236|24237|24238|24239|24240|24241|24242|24243|24244|24245| 24246|24247|24248|24249|24250|24251|24252|24253|24254|24255|24256|24257|24258|24259|24260|24261|323558|323559|323566|323567|323568|
323569|323581|323582|323595|323610|323651|323677|323678|923512|923533|923568|924077|924094|924106|924149|924156|924157|924158|924160|924162|924228)/i",$theData[$i][3]))
{
#$nyc[$i][0]="N.Y.C.";
/* $nyc[$i][0]=$theData[$i][0];
$nyc[$i][1]=$theData[$i][1];
$nyc[$i][2]=$theData[$i][4]; */
$nyc[]=explode($theData[$i][0],$theData[$i][1],$theData[$i][4]);
}
if (preg_match("/(23530|23531|23653|23659|23776|24019|24139|24193|24198|323649)/i",$theData[$i][3]))
{
#$millwood[$i][0]="MILWD";
$millwood[$i][0]=$theData[$i][0];
$millwood[$i][1]=$theData[$i][1];
$millwood[$i][2]=$theData[$i][4];
}
if (preg_match("/(23526|23586|23587|23588|23589|23590|23591|23592|23593|23595|23607|23608|23609|23610|23611|23612|23632|23639|23640|23641|
23642|23654|23754|23765|23769|24000|24148|24192|323565|323602|323613|323627|323648|923586|923587)/i",$theData[$i][3]))
{
#$hudvl[$i][0]="HUD VL";
$hudvl[$i][0]=$theData[$i][0];
$hudvl[$i][1]=$theData[$i][1];
$hudvl[$i][2]=$theData[$i][4];
}
if (preg_match("/(23655|24194|323650)/i",$theData[$i][3]))
{
#$dunwood[$i][0]="DUNWOD";
$dunwood[$i][0]=$theData[$i][0];
$dunwood[$i][1]=$theData[$i][1];
$dunwood[$i][2]=$theData[$i][4];
}
else {
unset($theData[$i]);
}
}
There is more to the code, but I guess it doesn't matter if i can't get past here without a timeout and/or out of memory. Yes i've tried increasing memory, it doesn't help.