Slow API response

[Dani]

Hi everyone, I really hope that you can help me diagnose an issue with my new API. I'm having the following issue:

I'm not happy with the performance of the app client at http://app.dazah.com that connects to it ... It uses cURL to issue about 3-4 API requests per page to https://www.dazah.com/api, and half the requests are always speedy, and the other half of the requests take a ridiculously long time. (Half take about 150 seconds per request and the other half take about 800 seconds per request, so every page takes at least 1.5-2 full seconds to load). If there is only a single request per page, it always take a really long time. The speed of the requests never has anything to do with the amount of computational resources required to serve the page. I'm convinced this is a networking issue. Even if all it's trying to do is fetch a page that spits out Hello World, it may take as much as 950 seconds.

Some requests are GET and some are POST. No difference in performance.

When the same API requests are made via my web browser, it's always speedy (about 250 ms per request). When they are made via the Postman Chrome extension, it's always speedy as well. I am in New York and the app.dazah.com server is in New Jersey. The API is hosted in Dallas, so it's not like my web browser is any closer to the API than the app is.

My web app is written in PHP and uses the cURL PHP wrapper. However, I do not get any better performance issuing a curl request from the command line.

What I did notice is that when I create an API endpoint that does a dump of $_SERVER[], I consistently get different results depending on whether the exact same request is initiated by a web browser (that takes about 150 ms to load the request) or cURL (that takes about 950 ms to load the request).

Chrome / Postman have HTTP_X_SSL_CIPHER set to a string, and have SERVER_PORT set to 80. cURL does not have HTTP_X_SSL_CIPHER, and has SERVER_PORT set to 443.

Why would two different clients cause this different behavior when trying to connect to the exact same endpoint?

This is what Chrome tells me when I load an endpoint at https://www.dazah.com that pulls up $_SERVER

array(35) {
  ["USER"]=>
  string(7) "daniweb"
  ["HOME"]=>
  string(13) "/home/daniweb"
  ["FCGI_ROLE"]=>
  string(9) "RESPONDER"
  ["SCRIPT_FILENAME"]=>
  string(31) "/usr/share/nginx/html/index.php"
  ["QUERY_STRING"]=>
  string(0) ""
  ["REQUEST_METHOD"]=>
  string(3) "GET"
  ["CONTENT_TYPE"]=>
  string(0) ""
  ["CONTENT_LENGTH"]=>
  string(0) ""
  ["SCRIPT_NAME"]=>
  string(10) "/index.php"
  ["REQUEST_URI"]=>
  string(12) "/welcome/foo"
  ["DOCUMENT_URI"]=>
  string(10) "/index.php"
  ["DOCUMENT_ROOT"]=>
  string(21) "/usr/share/nginx/html"
  ["SERVER_PROTOCOL"]=>
  string(8) "HTTP/1.1"
  ["GATEWAY_INTERFACE"]=>
  string(7) "CGI/1.1"
  ["SERVER_SOFTWARE"]=>
  string(12) "nginx/1.0.15"
  ["REMOTE_ADDR"]=>
  string(13) "10.143.27.250"
  ["REMOTE_PORT"]=>
  string(5) "34752"
  ["SERVER_ADDR"]=>
  string(13) "10.143.27.203"
  ["SERVER_PORT"]=>
  string(2) "80"
  ["SERVER_NAME"]=>
  string(13) "www.dazah.com"
  ["REDIRECT_STATUS"]=>
  string(3) "200"
  ["HTTP_HOST"]=>
  string(13) "www.dazah.com"
  ["HTTP_CONNECTION"]=>
  string(10) "keep-alive"
  ["HTTP_CACHE_CONTROL"]=>
  string(8) "no-cache"
  ["HTTP_USER_AGENT"]=>
  string(110) "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36"
  ["HTTP_POSTMAN_TOKEN"]=>
  string(36) "89d81689-6094-f2d9-162b-ddadf8b399a3"
  ["HTTP_ACCEPT"]=>
  string(3) "*/*"
  ["HTTP_ACCEPT_ENCODING"]=>
  string(23) "gzip, deflate, sdch, br"
  ["HTTP_ACCEPT_LANGUAGE"]=>
  string(14) "en-US,en;q=0.8"
  ["HTTP_COOKIE"]=>
  string(99) "dazah_csrf=a78d4fe9fc3e529ff5bd190d56d6ea1b; dazah_session=REMOVED"
  ["HTTP_X_SSL_CIPHER"]=>
  string(76) "AES128-GCM-SHA256       TLSv1.2 Kx=RSA      Au=RSA  Enc=AESGCM(128) Mac=AEAD"
  ["HTTP_X_FORWARDED_FOR"]=>
  string(13) "184.74.210.50"
  ["PHP_SELF"]=>
  string(10) "/index.php"
  ["REQUEST_TIME_FLOAT"]=>
  float(1469295148.9875)
  ["REQUEST_TIME"]=>
  int(1469295148)
}

And this is what the cURL request gives me:

string(1187) "array(26) {
  ["USER"]=>
  string(7) "daniweb"
  ["HOME"]=>
  string(13) "/home/daniweb"
  ["FCGI_ROLE"]=>
  string(9) "RESPONDER"
  ["SCRIPT_FILENAME"]=>
  string(31) "/usr/share/nginx/html/index.php"
  ["QUERY_STRING"]=>
  string(0) ""
  ["REQUEST_METHOD"]=>
  string(3) "GET"
  ["CONTENT_TYPE"]=>
  string(0) ""
  ["CONTENT_LENGTH"]=>
  string(0) ""
  ["SCRIPT_NAME"]=>
  string(10) "/index.php"
  ["REQUEST_URI"]=>
  string(12) "/welcome/foo"
  ["DOCUMENT_URI"]=>
  string(10) "/index.php"
  ["DOCUMENT_ROOT"]=>
  string(21) "/usr/share/nginx/html"
  ["SERVER_PROTOCOL"]=>
  string(8) "HTTP/1.1"
  ["GATEWAY_INTERFACE"]=>
  string(7) "CGI/1.1"
  ["SERVER_SOFTWARE"]=>
  string(12) "nginx/1.0.15"
  ["REMOTE_ADDR"]=>
  string(12) "45.33.82.238"
  ["REMOTE_PORT"]=>
  string(5) "39678"
  ["SERVER_ADDR"]=>
  string(13) "169.55.25.107"
  ["SERVER_PORT"]=>
  string(3) "443"
  ["SERVER_NAME"]=>
  string(13) "www.dazah.com"
  ["REDIRECT_STATUS"]=>
  string(3) "200"
  ["HTTP_HOST"]=>
  string(13) "www.dazah.com"
  ["HTTP_ACCEPT"]=>
  string(3) "*/*"
  ["PHP_SELF"]=>
  string(10) "/index.php"
  ["REQUEST_TIME_FLOAT"]=>
  float(1469295176.7073)
  ["REQUEST_TIME"]=>
  int(1469295176)
}

The first one took 126 ms. The second one took 1.18 seconds.

This is the cURL request

    $ch = curl_init('https://www.dazah.com/welcome/foo');
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($ch, CURLOPT_HTTPHEADER, array(
        'Connection' => 'keep-alive',
        'Cache-Control' => 'no-cache',
        'Accept' => '*/*',
        'Accept-Encoding' => 'gzip, deflate, sdch, br',
        'Accept-Language' => 'en-US,en;q=0.8',
        'User-Agent' => 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
    ));
    $response = (curl_exec($ch));

I was playing with trying to emualte adding User-Agent headers to try to force the same behavior as the web browser, but I guess I was doing something wrong. As you can see, I'm passying in Connection: keep-alive headers but they are not showing up in the $_SERVER dump.

[Dani]

This is what curl_getinfo() spit out for the above request:

array(26) {
  ["url"]=>
  string(33) "https://www.dazah.com/welcome/foo"
  ["content_type"]=>
  string(24) "text/html; charset=UTF-8"
  ["http_code"]=>
  int(200)
  ["header_size"]=>
  int(665)
  ["request_size"]=>
  int(63)
  ["filetime"]=>
  int(-1)
  ["ssl_verify_result"]=>
  int(0)
  ["redirect_count"]=>
  int(0)
  ["total_time"]=>
  float(0.498006)
  ["namelookup_time"]=>
  float(0.15438)
  ["connect_time"]=>
  float(0.198839)
  ["pretransfer_time"]=>
  float(0.406906)
  ["size_upload"]=>
  float(0)
  ["size_download"]=>
  float(1187)
  ["speed_download"]=>
  float(2383)
  ["speed_upload"]=>
  float(0)
  ["download_content_length"]=>
  float(-1)
  ["upload_content_length"]=>
  float(0)
  ["starttransfer_time"]=>
  float(0.493861)
  ["redirect_time"]=>
  float(0)
  ["certinfo"]=>
  array(0) {
  }
  ["primary_ip"]=>
  string(13) "169.55.25.107"
  ["primary_port"]=>
  int(443)
  ["local_ip"]=>
  string(12) "45.33.82.238"
  ["local_port"]=>
  int(41800)
  ["redirect_url"]=>
  string(0) ""
}

As you can see, 500 ms just to pull up a webpage that does nothing other than retrieve a dump of $_SERVER.

To put that into perspective, Chrome Developer Tools told me that retrieving this particular forum thread, which required 5 database queries in addition to all sorts of computation processing, was delivered in only 137 ms. (.15 ms to send the request, 133.69 ms waiting, and 2.84 ms downloading the HTML).

[cereal]

Hi Dani!

You can see the response header and the request flow by adding:

curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);

I'm passying in Connection: keep-alive headers but they are not showing up in the $_SERVER dump.

It probably happens because you're submitting the headers as an indexed array, instead it should be:

curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    'Connection: keep-alive',
    'Cache-Control: no-cache',
    'Accept: */*',
    'Accept-Encoding: gzip, deflate, sdch, br',
    'Accept-Language: en-US,en;q=0.8',
    'User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'
));

Bye!

[rubberman]

Also, a lot of web sites use multiple servers behind a "load balancer", most of which such as F5 devices, are round-robin and it is entirely possible for one request to get passed to a server under low-load, but the next to a server that is grinding away on other jobs. We certainly had this issue at Nokia with our proxy servers. You might also want to consider Cereal's suggestion about the array encoding.

[rubberman]

BTW, what version of PHP are you using for this project? And did you install as a package, or build it from source?

[Dani]

It probably happens because you're submitting the headers as an indexed array

Well that's ridiculously counter-intuitive! Still didn't fix the problem though, I'm afraid.

Also, a lot of web sites use multiple servers behind a "load balancer", most of which such as F5 devices, are round-robin and it is entirely possible for one request to get passed to a server under low-load, but the next to a server that is grinding away on other jobs.

That's not the case here because, even though we are behind a firewall and load balancer, I am in control of both the server and the client.

BTW, what version of PHP are you using for this project? And did you install as a package, or build it from source?

This is the client:

https://github.com/Dani28/Dazah-Business-Networking

It's installed locally. Check it out!!!!! ;)

P.S. After you've checked it out, help me figure out what's wrong.

[rproffitt]

I'm probably way off but do try using IP address instead of domain name. This one dogged a friends testing for months.

Again, just something I've run into.