Member Avatar for khan17

hello all,

pls have some patience to read this and help me.

Im a student and doing a project using CSharp. Im doing an automation process. For which i ve to login to a website click some links and then download files from there. I cant download all the files there. The downloading also done with some conditions. so i need to get the details from the website and store in a database table. And then i need to perform operations something like ,

if (table2 3rd column element equals table3 3rd column element)
then download a particular file from the web page


Till now i can navigate, login, perform clicks and then i can extract the table datas from the web page which is need to be stored in sql database table. I ve presently keeping the extracted data from web page in a text files. can i transfer the data in text file to sql to be in table format not as a whole textfile.

or else is there any way to store directly the extracted content in sql?

am i going in a right direction?

if not means pls guide me..

thanks..

  • 2 Contributors
  • 1 Reply
  • 126 Views
  • 9 Hours Discussion Span
  • Latest Post Latest Post by mcriscolo

Dani AI

Generated

Short answer: yes — keep moving toward storing structured rows in SQL rather than a single text blob. was right to push for parsing into fields. The practical goal is: automate the site, turn each HTML table row into a typed record, persist those records, then run the comparison/download logic against the persisted rows.

A simple, reliable workflow:

  • Automate and authenticate (use Selenium WebDriver for JS-heavy pages or HttpClient for direct requests).
  • Parse the page into C# objects (Html Agility Pack is great for extracting table rows).
  • Persist rows into a well-designed table (columns for the key values you compare, plus metadata: source URL, download_status, last_checked, local_path, checksum).
  • Evaluate your condition (SQL join or a small in-memory LINQ/SQL query) and queue/download matching files.

For inserting data: use parameterized inserts for small batches and SqlBulkCopy for larger imports. Example pattern for bulk load:

var table = new DataTable();
table.Columns.Add("Key", typeof(string));
table.Columns.Add("ColA", typeof(string));
table.Columns.Add("ColB", typeof(string));
// populate rows...

using(var bulk = new SqlBulkCopy(connString))
{
    bulk.DestinationTableName = "dbo.ExtractedRows";
    bulk.WriteToServer(table);
}

For single-row inserts:

using(var cmd = new SqlCommand("INSERT INTO ExtractedRows (Key, ColA, ColB) VALUES (@k,@a,@b)", conn))
{
    cmd.Parameters.AddWithValue("@k", key);
    cmd.Parameters.AddWithValue("@a", colA);
    cmd.Parameters.AddWithValue("@b", colB);
    cmd.ExecuteNonQuery();
}

Important cautions and tips: honor the site’s terms of use and rate limits; preserve cookies/session headers when downloading; add retries and exponential backoff; use a unique constraint or checksum to avoid duplicate downloads; log failures so you can reprocess later. Useful references: , Selenium WebDriver, and the SqlBulkCopy docs.

Member Avatar for mcriscolo

can i transfer the data in text file to sql to be in table format not as a whole textfile.

That depends on how you need to access the data afterward. You could shove the entire text blob into a database table as one column, or, if you've split out the extracted data into fields (like name, address, phone number, etc.), you could set up a database table with columns for each of those fields, then insert a row containing that data.

Usually, it's better up front to parse and cleanup the data you've extracted, separate into fields, and then setup a table that holds that data. Makes the "downstream" process of using the extracted data easier.

Does that help?

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.