Curl Posting thousands of looped rows at once in php

[sammry]

This is a SMS DLR application from http://www.smsgatewaycenter.com which is sent by them to my client's url and I need to configure for my client the same from my client's url to his reseller's url. This script uses every 1 hour to post data to my client's reseller/customer. So, it piles up with thousands of report to send.

I am trying to send dataset which is more than 10000 looped rows using curl. I have tried with curl_multi_exec and curl_setopt_array. No matter what, it sends out one by one which is causing load on server. Is there a way, where I can send/forward set of desired request

what i have tried so far for curl_multi_exec

            $query = $db->execute("SELECT * FROM `dlr_tbl` WHERE `fetched` = '0'");
            $row = $query->getrows();
            for($i=0;$i<count($row);$i++) {
                $extid = $row[$i]['external_id'];
                $status = $row[$i]['delivery_status'];
                $cause = $row[$i]['cause'];
                $mobile = $row[$i]['mobile_no'];
                $date = $row[$i]['delivery_date'];
                $requestarray = "externalId=".$extid."&status=".$status."&cause=".$cause."&phoneNo=".$mobile."&date=".$date;
                $furl = "http://www.example.com/library/getresponse.php?".$requestarray;
                $ch1 = curl_init();
                curl_setopt($ch1, CURLOPT_URL, $furl);
                curl_setopt($ch1, CURLOPT_HEADER, 0);
                $mh = curl_multi_init();
                curl_multi_add_handle($mh,$ch1);
                $active = null;
                do {
                    $mrc = curl_multi_exec($mh, $active);
                }

                while ($mrc == CURLM_CALL_MULTI_PERFORM);
                while ($active && $mrc == CURLM_OK) {

                    if (curl_multi_select($mh) != -1) {
                        do {
                            $mrc = curl_multi_exec($mh, $active);
                        }

                        while ($mrc == CURLM_CALL_MULTI_PERFORM);
                    }

                }

                curl_multi_remove_handle($mh, $ch1);
                curl_multi_close($mh);
            }

and for curl_setopt_array

        $query = $db->execute("SELECT * FROM `dlr_tbl` WHERE `fetched` = '0'");
        $row = $query->getrows();
        for($i=0;$i<count($row);$i++) {
            $extid = $row[$i]['external_id'];
            $status = $row[$i]['delivery_status'];
            $cause = $row[$i]['cause'];
            $mobile = $row[$i]['mobile_no'];
            $date = $row[$i]['delivery_date'];
            $requestarray = "externalId=".$extid."&status=".$status."&cause=".$cause."&phoneNo=".$mobile."&date=".$date;
            $furl = "http://www.example.com/library/getresponse.php";
            $ch = curl_init($furl);
            curl_setopt_array($ch, array(     CURLOPT_URL => $furl,     CURLOPT_RETURNTRANSFER => true,     CURLOPT_POST => true,     CURLOPT_POSTFIELDS => $requestarray     ));
            $curl_scraped_page = curl_exec($ch);
            curl_close($ch);
        }

Is there a better way to do this for multiple rows or should I stick with it?
And also, smsgatewaycenter.com sends the data, the same way... but they send in multiple, more than 50,000 rows gets dumped into mysql table in less time. Where as my code takes a lot of time.

Please suggest or recommend the better way.

Thanks in advance.

[gabrielcastillo]

What does their API allow for? Can you send a json string the the server? If so, you could complie you dataset into a json string and send in one transaction.

[sammry]

It is what we send, we can include whatever we need, currently am looking at CURL POSTing, so it will be easier for all users...

[gabrielcastillo]

So why not combine all your rows into a json array and send them the server in one post request.

$request = json_encode(array('something' => $result['something'], 'something' => $result['something']));

[sammry]

Am not good with json, I can encode but nil knowledge when decoding at remote server. I searched online for json but nothing hit me.

Again, if it's more than 80000 rows then the url will be long enough, will there be a chance of losing json encoded data?

Please help me with the code to decode at remote server.

Thanks again for your responses...

[gabrielcastillo]

You can decode json by using the json_decode(); function. Docs

[Isaac_4]

Also all the data in the CURLOPT_POSTFIELDS will not be in the URL but rather part of the POST body, so don't worry about problems with URL length. gabrielcastillo's suggestion is a good way to do this.

[Isaac_4]

Some sample code. Post data to your customer's PHP file like this:

// Read data from DB
$query = $db->execute("SELECT * FROM `dlr_tbl` WHERE `fetched` = '0'");
$row = $query->getrows();

// Create an array of rows to encode, using the field names we want
$array_data = array();
for($i=0;$i<count($row);$i++) {
    $post_data[] = array(
        'extid' => $row[$i]['external_id'],
        'status' => $row[$i]['delivery_status'],
        'cause' => $row[$i]['cause'],
        'mobile' => $row[$i]['mobile_no'],
        'date' => $row[$i]['delivery_date'],
    );
}

// Encode data to a JSON packet
$json_data = json_encode($array_data);

echo 'DEBUG SEND: '.$json_data.'<hr/>'; // DEBUG only

// Post the JSON packet - no GET parameters needed, everything is
// in $json_data already
$furl = "http://www.example.com/library/getresponse.php";
$ch = curl_init($furl);
curl_setopt_array($ch, array(
    CURLOPT_URL => $furl,
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => $json_data
));
$curl_scraped_page = curl_exec($ch);
echo 'DEBUG RESULT: '.$curl_scraped_page; // DEBUG only
curl_close($ch);

Then in your customer's PHP file, getresponse.php, you can get the array back out like this:

// php://input returns all POST data. We want to decode it as JSON
$post_data = file_get_contents("php://input");
if($post_data) {
    if(substr($post_data, 1, 1) != '{') || substr($post_data, -1, 1) != '{')) exit('Error, POST data does not look like JSON');

    echo 'DEBUG RECV: '.$post_data.'<hr/>'; // DEBUG only
    $array_data = json_decode($post_data);
    var_dump($array_data); // DEBUG only
} else exit('Error, no data in POST!');

[sammry]

Thank you so much Isaac 4 and ardav.

Your code had some errors, hence I coded all new based on your concept and its working like a charm.

By the way, sorry for the later reply, as was not available online due to festival.

Thanks again.

[Isaac_4]

Yeah sorry about the errors, wasn't able to fully test the idea. Glad it helped though!