LibCurl - Copy webpage contents to String

[icewolf]

Hi There.

I am trying to use libCurl to copy the contents of a webpage into a string in C.

This is what I have so far:

void Cmd_translate_f (char *message)
{
	CURL *curl;
	CURLcode res;
	
	curl = curl_easy_init();
	if(curl) {
	curl_easy_setopt(curl, CURLOPT_URL, va("http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=%s!&langpair=en|fr"),message);
	curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data); 
	res = curl_easy_perform(curl);
	}

}

The function takes a string and then uses the google translating service to translate it. I am, however, unaware of how to copy the contents of the received webpage into a c string. All this code does at the moment is print out the contents.

Many Thanks :)

[nigecus]

i have the same question. technically i want to fetch .csv files from an url with libcurl. yes it works: after compiling the terminal print the .csv content.

however i have no clue how to get the "res" variable (CURLcode) into a string that might be treated further.

it would be perfect if the "res" thing could read by ifstream or something (to use getline or something). at the end of the day, i want to put parts of the .csv web content into a sql query string.

[jephthah]

I've never used libcurl. I'd love to help you, but all i can do is read the libcurl example that does exactly what you're looking for, and repeat what it says, but that would be kind of dumb and a waste of everyone's time, since you can just go read it yourself.

[nigecus]

try this:

/*  	g++ -c  -I/usr/include/curl testcurl.c
 *  	g++ -o testtest testcurl.o -L/usr/lib/libcurl -lcurl
 * 		./testtest
*/

#include <fstream>
#include <vector>
#include <iostream>
#include <sstream>
#include <string>
#include <stdio.h>
#include "/usr/include/curl/curl.h"
#include "/usr/include/curl/easy.h"

using namespace std;


int writer(char *data, size_t size, size_t nmemb, string *buffer){
	fprintf(stderr,"Hello I am a function pointer\n");
	int result = 0;
	if(buffer != NULL) {
		buffer -> append(data, size * nmemb);
		result = size * nmemb;
	}
	return result;
} 



int main ()
{
	/* (A) Variable Declaration */
	CURL *curl;		/* That is the connection, see: curl_easy_init */
	/*CURLcode res;		/* Not needed, see: curl_easy_cleanup */
	string buffer;		/* See: CURLOPT_WRITEDATA */
	
	/* (B) Initilise web query */
	curl = curl_easy_init();
	
	/* (C) Set Options of the web query 
	 * See also:  http://curl.haxx.se/libcurl/c/curl_easy_setopt.html  */
	if (curl){
		curl_easy_setopt(curl, CURLOPT_URL, "http://ichart.finance.yahoo.com/table.csv?s=DAI.DE&a=NaN&b=02&c=pr-2&g=d&ignore=.csv");
		curl_easy_setopt(curl, CURLOPT_HEADER, 0);	 /* No we don't need the Header of the web content. Set to 0 and curl ignores the first line */
		curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 0); /* Don't follow anything else than the particular url requested*/
		curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, writer);	/* Function Pointer "writer" manages the required buffer size */
		curl_easy_setopt(curl, CURLOPT_WRITEDATA, &buffer ); /* Data Pointer &buffer stores downloaded web content */	
	}
	else{
		fprintf(stderr,"Error 1\n");	/* Badly written error message */
		return 0;											
	}
	
	/* (D) Fetch the data */
    curl_easy_perform(curl);
	/* res = curl_easy_perform(curl); */
	/* There is no need for "res" as it is just a flag */	
	
	/* (E) Close the connection */
	curl_easy_cleanup(curl);
	
	/* (F) Transform &buffer into a istringstream object */
	std::istringstream iss(buffer);
	
	string line, item;	
    int linenum = 0;
    while (getline (iss, line)){
        linenum++;													/* Move to Next Line */
        cout << "\nLine #" << linenum << ":" << endl;				/* Terminal Printout */
        std::istringstream linestream(line);						/* Read Next Line */

        int itemnum = 0;
        while (getline (linestream, item, ',')){
            itemnum++;
            cout << "Item #" << itemnum << ": " << item << endl;	/* Terminal Printout */
        } // End WHILE (items)
    } //End WHILE (lines)
	
	return 0;	
}

[icewolf]

Eventually got it working.

size_t write_data(void *ptr, size_t size, size_t nmemb, FILE *stream) {
    size_t written;
    written = fwrite(ptr, size, nmemb, stream);
    return written;
}

void FuncBlahBlah {
        curl = curl_easy_init();
	if(curl) {
	fp = fopen("C:\\trans.txt","wb");
	curl_easy_setopt(curl, CURLOPT_URL, "http://ajax.googleapis.com/ajax/services/language/translate?v=1.0&q=hello&langpair=en|fr");
	curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, write_data); 
	curl_easy_setopt(curl, CURLOPT_WRITEDATA, fp);
	res = curl_easy_perform(curl);
	curl_easy_cleanup(curl);
    fclose(fp);
	}
	fp = fopen("C:\\trans.txt","rb");
	fseek (fp , 0 , SEEK_END);
	lSize = ftell (fp);
	rewind (fp);
	fread (translated,1,lSize,fp);
	translated[lSize] = 0;
	fclose(fp);
}

The only way i could get it to work was to save the output to a file and re-read it and put it in a string. It does work great though. Hope this helps.