Member Avatar for bcm

I want to retrive data for this path which i can do, but the problem is I want the lastpart i.e. 1378 to increase by one every time

""

How can I do that in VB.NET2003???

  • 3 Contributors
  • 2 Replies
  • 98 Views
  • 9 Hours Discussion Span
  • Latest Post Latest Post by binoj_daniel

Dani AI

Generated

— the simplest model is to generate the next numeric id and append it to the URL; pointed this out and correctly suggested extracting the existing id if you need to continue from a given URL. Below is a compact, VB.NET 2003 (CLR 1.1) friendly pattern that (a) pulls a trailing number from an example URL, (b) iterates ids, (c) fetches each page using HttpWebRequest, and (d) handles common errors and politeness concerns.

' get starting id from an existing URL (uses a regex to capture trailing digits)
Dim startId As Integer = 1000
Dim m As System.Text.RegularExpressions.Match = _
    System.Text.RegularExpressions.Regex.Match(someUrlString, "(\d+)(?=\.html$)")
If m.Success Then startId = Integer.Parse(m.Groups(1).Value)

Dim endId As Integer = startId + 100

For id As Integer = startId To endId
    Dim url As String = "http://example.com/prefix_" & id.ToString() & ".html"
    Dim req As System.Net.HttpWebRequest = _
        CType(System.Net.HttpWebRequest.Create(url), System.Net.HttpWebRequest)
    req.Timeout = 15000
    req.UserAgent = "Mozilla/4.0 (compatible)"
    Try
        Dim resp As System.Net.HttpWebResponse = _
            CType(req.GetResponse(), System.Net.HttpWebResponse)
        Dim sr As New System.IO.StreamReader(resp.GetResponseStream())
        Dim html As String = sr.ReadToEnd()
        sr.Close()
        resp.Close()
        ' process html here (write to disk/db to avoid memory spikes)
    Catch ex As System.Net.WebException
        If ex.Response IsNot Nothing Then
            CType(ex.Response, System.Net.HttpWebResponse).Close()
        End If
    End Try
    System.Threading.Thread.Sleep(250) ' throttle requests
Next

Notes and cautions: on VS2003/.NET 1.1 use HttpWebRequest (WebClient arrived in .NET 2.0). Always set a User-Agent and timeout, close streams/responses, catch WebException to detect 404s, and throttle (Thread.Sleep or a timer) so the target server is not hammered. For UI apps run the fetch on a worker thread to avoid freezing the form, and write results incrementally to disk or DB if collecting many pages.

Recommended Answers

All 2 Replies

Member Avatar for hollystyles 
Dim MIN As Integer = 1378
Dim MAX As Integer = 2000

For i As Integer = MIN To Max
        SomeSub(String.Format("", i.ToString()))

Next i

Private Sub SomeSub(url As String)
    'Do some funky thing with url
End Sub

Or something like that... gad I hate VB syntax, yuck!

Member Avatar for binoj_daniel

Use InStrRev function to get the position of the "_" and then use left$ to get the required text.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.