Downloading web page source.

[Eddy Dean]

Hello everyone,

I 've been looking for a solution for this problem for some time now but I don't seem to figure it out.

I want to download the source of a web page (the HTML source). I found several API function that have to do something with internet and downloading for example InternetOpen, InternetOpenUrl .

This didn't work. I tried almost everything but it wouldn't compile.

I found this code on MSDN:

int main(int argc, char* argv[])
{
	HINTERNET hNet = ::InternetOpen("MSDN SurfBear",
                                PRE_CONFIG_INTERNET_ACCESS,
                                NULL,
                                INTERNET_INVALID_PORT_NUMBER,
                                0) ;

HINTERNET hUrlFile = ::InternetOpenUrl(hNet,
                                "http://www.microsoft.com",
                                NULL,
                                0,
                                INTERNET_FLAG_RELOAD,
                                0) ;

char buffer[10*1024] ;
DWORD dwBytesRead = 0;
BOOL bRead = ::InternetReadFile(hUrlFile,
                                buffer,
                                sizeof(buffer),
                                &dwBytesRead);

::InternetCloseHandle(hUrlFile) ;

::InternetCloseHandle(hNet) ;
	return 0;
}

It keeps giving me the following errors:

--------------------Configuration: internetopen - Win32 Debug--------------------
Linking...
internetopen.obj : error LNK2001: unresolved external symbol __imp__InternetCloseHandle@4
internetopen.obj : error LNK2001: unresolved external symbol __imp__InternetReadFile@16
internetopen.obj : error LNK2001: unresolved external symbol __imp__InternetOpenUrlA@24
internetopen.obj : error LNK2001: unresolved external symbol __imp__InternetOpenA@20
Debug/internetopen.exe : fatal error LNK1120: 4 unresolved externals
Error executing link.exe.

internetopen.exe - 5 error(s), 0 warning(s)

One project I downloaded does work. A bit. It downloads the first several lines of the source and then just outputs fffffffffffffffffffffffffffffffff (it the buffer gets filler with ff-s after the first lines of source).

Also this is an MFC project, and I prefer not using MFC.

I'll post the MFC code below:

void CMFCinternetDlg::OnButton1() 
{
	// TODO: Add your control notification handler code here
	CInternetSession* pInternetSession;
	pInternetSession = new CInternetSession();

	if (!pInternetSession)
	{
		MessageBox("Could not establish HTTP session, please check your connection","error");
		return;
	}

	CStdioFile* pFile = NULL;

	char* buffer;
	buffer = new char[30000];
	
	pFile = pInternetSession->OpenURL(CString("http://www.my-url.com"));

	pFile->Read(buffer, 30000);

	ofstream myfile;
	myfile.open ("example.txt");
	myfile << buffer;
	myfile.close();


	pFile->Close();
	pInternetSession->Close();
	
}

Well, this is what I've tried so far, but as I told before no succes.

I would greatly appreciate it if anyone could point me in the right direction.

Greetz, Eddy

[GloriousEremite]

For your linker errors, you need to link with wininet.lib. Also, I don't think InternetReadFile null-terminates the data, so take that into account.

[~s.o.s~]

I dont know a lot about handling internet protocols using C++ and MFC but i think this should help you a bit.

http://www.codeguru.com/cpp/i-n/internet/generalinternet/article.php/c3411/

Waiting for your feedback.
Bye.

[Eddy Dean]

This is what I've got so far. As you can see I added some error checking and the errors are at HttpSendRequest(hHttpFile, NULL, 0, NULL, 0); I'll post the full source here:

// openmic.cpp : Defines the entry point for the console application.
//

#include "stdafx.h"
#include <stdio.h>
#include <iostream>
#include <windows.h>
#include <wininet.h>


int main(int argc, char* argv[])
{
	// Open Internet session.
HINTERNET hSession = ::InternetOpen("My app name",
                                    PRE_CONFIG_INTERNET_ACCESS,
                                    NULL, 
                                    INTERNET_INVALID_PORT_NUMBER,
                                    0) ;
if(!hSession){printf("Error while opening the internet session \n"); return 0;}

// Connect to www.microsoft.com.
HINTERNET hConnect = ::InternetConnect(hSession,
                                    "http://www.microsoft.com",
                                    INTERNET_INVALID_PORT_NUMBER,
                                    "",
                                    "",
                                    INTERNET_SERVICE_HTTP,
                                    0,
                                    0) ;
if(!hConnect){printf("Error while connecting to the server \n"); return 0;}
// Request the file /MSDN/MSDNINFO/ from the server.
HINTERNET hHttpFile = ::HttpOpenRequest(hConnect,
                                     NULL,
                                     "/MSDN/MSDNINFO/",
                                     HTTP_VERSION,
                                     NULL,
                                     0,
                                     INTERNET_FLAG_DONT_CACHE,
                                     0) ;
if(!hHttpFile){printf("Error while requesting the file \n"); return 0;}

// Send the request.
BOOL bSendRequest = ::HttpSendRequest(hHttpFile, NULL, 0, NULL, 0);

if(!bSendRequest){ printf("Error while sending request \n"); return 0;}

// Allocate a buffer for the file.   
char* buffer = new char[50000] ;

// Read the file into the buffer. 
DWORD dwBytesRead ;
BOOL bRead = ::InternetReadFile(hHttpFile,
                                buffer,
                                3000, 
                                &dwBytesRead);
// Put a zero on the end of the buffer.
buffer[dwBytesRead] = 0 ;

if(!bRead) buffer = "Nothing has been loaded";
// Close all of the Internet handles.
::InternetCloseHandle(hHttpFile); 
::InternetCloseHandle(hConnect) ;
::InternetCloseHandle(hSession) ;


printf("buffer: %s", buffer);
	return 0;
}

@everyone: thanks for your support, you've been very helpfull.

Greetz, Eddy

[~s.o.s~]

It would be better if you describe the error you get or print the error report here since all of us dont have Wininet library.

Yours truly,
~s.o.s~

[Eddy Dean]

Sorry if I didn't explain well. I'm not getting any errors while compiling or building. The buffer was just empty so I added error checking , like if(!bSendRequest){ printf("Error while sending request \n"); return 0;} now when I run the program I get "Error while sending request" as output to the command.

This made me think that somewhere went wrong at that function.

Greetz, Eddy

[~s.o.s~]

Try this instead of your stuff and see if this works.

// m_strURL => ur url
 
HINTERNET hINet, hFile;
hINet = InternetOpen("InetURL/1.0", INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0 );
if ( !hINet )
{
// bail out
}
hFile = InternetOpenUrl( hINet, "m_strURL, NULL, 0, 0, 0 );
if ( hFile )
{
CHAR buffer[1024];
DWORD dwRead;
while ( InternetReadFile( hFile, buffer, 1023, &dwRead ) )
{
if ( dwRead == 0 )
     break;
buffer[dwRead] = 0;
m_strURL += buffer;
}
InternetCloseHandle( hFile );
}
InternetCloseHandle( hINet );

[Eddy Dean]

huhhwwhaddayado?! (translation: Huh? What did you do?)

It works! It really does!

Thanks a million!

Greetz, Eddy

[~s.o.s~]

Glad i could help, after all this is why we are all here, right. ;)

And it would be really nice if you could post your entire working code along with everything so that it would be a good reference for other ppl.

[Eddy Dean]

You pretty much posted the whole code. I don't think it's usefull to post the full source of my program because I've got 250 lines so far (I don't know if comment is counted too, can anyone verify this?)

I guess that this project will be ~500 lines when it's finished.

Greetz,Eddy

[~s.o.s~]

Maybe as an attachment, when your project is finished so that a newbie can just copy and paste the code and understand its working. Of course if there are some other issues or reasons you dont want to post code then its ok.

Hoping to hear from you.

Regards,
~s.o.s~