Member Avatar for 100110010110100

I have tried parsing a website with curl and then searching it with regular expressions, but I have not been able to perform a regex search. Even if I try parsing news.google.com and then search for any non digit, the code returns that there is no match.

If I run a regex search on a simple inputted string, on the other hand, it does find a match with no problem.

<?

//source website
$source = "http://news.google.com/?topic=w";

//function to parse with cURL
function openWithCURL($url){
 	$curl_handle=curl_init();
	curl_setopt($curl_handle,CURLOPT_URL, $url);
	$result = curl_exec($curl_handle);
	curl_close($curl_handle);
}


// parse website
$parsedsite = openWithCURL($source);

//Regular Expression to search for any single non digit (made it very inclusive so as to test out whether the regular expression works)

$regexp="/\D/";

if (preg_match($regexp, $parsedsite, $matches)) {
	echo "Found it!!";
	echo $matches[0];
}
	else {
	echo "Didn't find it!!";
}

?>

Any assistance in determining why I cannot seem to use regular expressions on information parsed from a webpage using curl would be greatly appreciated. Thanks.

  • 2 Contributors
  • 2 Replies
  • 928 Views
  • 8 Hours Discussion Span
  • Latest Post Latest Post by 100110010110100

Dani AI

Generated

Short answer: the regex is almost certainly being run against an empty or wrong string. Two likely causes in this thread are (a) the cURL call is not returning the page body (Insensus already flagged that your function needed to return the result) and (b) the value passed to cURL is an HTML anchor (or contains unescaped quotes) instead of a plain URL, so nothing valid is fetched.

Quick checklist to resolve and debug:

  • Confirm the fetch function sets CURLOPT_RETURNTRANSFER so curl_exec returns the HTML instead of echoing it.
  • Verify you are passing a URL string (e.g. "http://example.com") — not an <a ...> tag — and that PHP string quotes are correct/escaped.
  • Immediately after fetching, inspect the value: use var_dump($parsedsite) and echo substr($parsedsite,0,200) to see what you actually got.
  • If fetch fails, check curl_error() / curl_getinfo(); if preg_match fails unexpectedly, check preg_last_error().

A safe fetch pattern (example) that avoids the common pitfalls:

function fetch_url($url) {
    $ch = curl_init();
    curl_setopt_array($ch, array(
        CURLOPT_URL => $url,
        CURLOPT_RETURNTRANSFER => true,
        CURLOPT_FOLLOWLOCATION => true,
        CURLOPT_USERAGENT => 'Mozilla/5.0 (PHP)',
        CURLOPT_TIMEOUT => 10,
    ));
    $body = curl_exec($ch);
    if ($body === false) {
        $err = curl_error($ch);
        curl_close($ch);
        throw new RuntimeException('cURL error: '.$err);
    }
    curl_close($ch);
    return $body;
}

For HTML work: avoid brittle regex on full pages. Use DOMDocument/DOMXPath to extract elements, then run simple string or regex checks on the extracted text or attributes. After fixing return/URL/quoting issues and confirming the fetched string, preg_match('/\D/', $string) will behave as expected.

Recommended Answers

All 2 Replies

Member Avatar for Insensus

Your openWithCURL function doesn't return anything.
I suppose you want

function openWithCURL($url){
 	$curl_handle=curl_init();
	curl_setopt($curl_handle,CURLOPT_URL, $url);
	$result = curl_exec($curl_handle);
	curl_close($curl_handle);
	return $result;
}
Member Avatar for 100110010110100

Thanks, I had originally had the function just as you say, set to

return $results

. However, I just tried modifying the script to add the return value back in, and the regular expression still doesn't work.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.