Member Avatar for mahmoud209

Hi everyone,
i was working on a project lately where my manager asked me to develop a vb application that can download files of the website
<a href="http://apps1.eere.energy.gov/buildings/energyplus/cfm/weather_data.cfm">http://apps1.eere.energy.gov/buildings/energyplus/cfm/weather_data.cfm</a>[<a href="http://apps1.eere.energy.gov/buildings/energyplus/cfm/weather_data.cfm" target="_blank" title="New Window">^</a>]
Internally.
i was thinking if i can make a form that display all downloadable files on the website with a checkBox next to every file so that the user can check which files he wants to download and proceed.
please help I’ve been searching the web for last three days and can't find anything

  • 3 Contributors
  • 3 Replies
  • 291 Views
  • 1 Day Discussion Span
  • Latest Post Latest Post by GeekByChoiCe

Dani AI

Generated

A practical, maintainable approach depends on whether you can change the server. If you can, a server-side manifest or an API that returns the directory listing is the cleanest and safest option (echoing ’s suggestion). If you cannot change the server, build a small, robust crawler in the client app that extracts candidate links, validates them, and then presents them with checkboxes for the user (building on the idea from and the downloading shown by ).

Basic workflow to implement

  1. Fetch the page HTML.
  2. Parse anchor tags with an HTML parser (avoid brittle regex).
  3. Normalize relative URLs with the Uri class.
  4. Filter by known file extensions and/or issue a HEAD request to check Content-Type or Content-Disposition for attachment filenames.
  5. De-duplicate, show results in a checkbox list, then download selected items with resumable/streaming downloads and progress reporting.

Example VB.NET snippet (uses HttpClient + HtmlAgilityPack)

' Requires HtmlAgilityPack (NuGet)
Imports System.Net.Http
Imports HtmlAgilityPack
Imports System.IO

Async Function GetDownloadableLinksAsync(pageUrl As String) As Task(Of List(Of Uri))
    Dim results As New List(Of Uri)
    Using client As New HttpClient()
        Dim html = Await client.GetStringAsync(pageUrl)
        Dim doc As New HtmlDocument()
        doc.LoadHtml(html)
        Dim anchors = doc.DocumentNode.SelectNodes("//a[@href]")
        If anchors Is Nothing Then Return results
        For Each a In anchors
            Dim href = a.GetAttributeValue("href", String.Empty).Trim()
            If String.IsNullOrEmpty(href) Then Continue For
            Dim resolved = New Uri(New Uri(pageUrl), href)
            Dim ext = Path.GetExtension(resolved.AbsolutePath).ToLowerInvariant()
            If New String() {".zip", ".pdf", ".csv", ".epw", ".exe"}.Contains(ext) Then
                results.Add(resolved)
            Else
                Dim head = New HttpRequestMessage(HttpMethod.Head, resolved)
                Dim resp = Await client.SendAsync(head)
                Dim disp = If(resp.Content?.Headers?.ContentDisposition, Nothing)
                If resp.IsSuccessStatusCode AndAlso disp IsNot Nothing AndAlso Not String.IsNullOrEmpty(disp.FileName) Then
                    results.Add(resolved)
                End If
            End If
        Next
    End Using
    Return results
End Function

Troubleshooting and cautions

  • Respect robots.txt and the site terms of use. Scraping without permission can be blocked or illegal.
  • Handle redirects, authentication, cookies, SSL cert issues, rate limits and backoff.
  • For large or recurring jobs, use a server-side manifest or a small DB (as suggested) to avoid repeated scraping and to keep state (downloaded files, timestamps, hashes).

Recommended Answers

All 3 Replies

Member Avatar for JJCollins

I'm going to guess that you're not familiar with VB much?

Here is some sample code to actually Ping/Download/Upload from a website

If My.Computer.Network.Ping("www.anywebsite.com", 1000) Then
  		MsgBox("Server pinged successfully.")
	Else
  		MsgBox("Ping request timed out.")
	End If

	My.Computer.Network.DownloadFile _
    	("", _
    	"C:\Documents\anyfile.txt")

	My.Computer.Network.UploadFile( _
	"C:\Documents\anyfile.txt", _
	"")

Obviously you can define what folder to save to/upload from etc, hope this helps.

As for listing the files from the website, i've never had to do it. Personally, I would look at creating a small database (if possible) to hold information about the downloadable files on the system with a URL field pointing to the location of the file which can be selected via a database query and the URL returned to the user or even redirected automatically on the click of a button.

- Jordan

Member Avatar for mahmoud209

Thanks Jordan, i'm still thinking of more ideas perhaps something like reading the html source code of the page and pick the Links and identify the downloadable files. But your idea also is a great help. thanks

Member Avatar for GeekByChoiCe

you could store a php file on the server which returns the list of files which are in a given directory and requesting that list with your application.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.