Member Avatar for khan17

hello all,

i want to save the web page content into a database. how can i fetch data from that website and save into a sql database. im using c# as front end.

need help.

expecting reply soon since it needed for my project.

thanks.

  • 2 Contributors
  • 5 Replies
  • 133 Views
  • 2 Days Discussion Span
  • Latest Post Latest Post by Mohtshm

Dani AI

Generated

wanted to save a web page table into SQL Server and suggested capturing rendered HTML. That can work for the page your app itself renders, but for pulling a table from another site the robust pattern is: fetch the page HTML, parse the DOM to extract the table, convert rows to a tabular structure, then persist to SQL (avoid fragile substring/indexOf hacks).

A concise workflow (C#):

  • Use HttpClient to GET the page (or a headless browser if the table is populated by JavaScript).
  • Parse HTML with a DOM parser like HtmlAgilityPack and select the desired <table>.
  • Build a DataTable (or a list of strongly typed objects).
  • Persist with parameterized inserts or SqlBulkCopy for bulk loads.

Example (outline):

using var http = new HttpClient();
var html = await http.GetStringAsync(url);

var doc = new HtmlAgilityPack.HtmlDocument();
doc.LoadHtml(html);
var table = doc.DocumentNode.SelectSingleNode("//table");
if (table == null) return;

var dt = new DataTable();
// build columns from header row then add data rows...
// finally bulk insert:
using var conn = new SqlConnection(connString);
conn.Open();
using var bcp = new SqlBulkCopy(conn);
bcp.DestinationTableName = "TargetTable";
bcp.WriteToServer(dt);

Notes and troubleshooting:

  • If the page uses client-side JS to render the table, HttpClient will not see it. Use Playwright or Selenium to get the post-JS DOM.
  • Normalize encoding and HTML entities (HtmlDecode) before inserting.
  • For unstable column sets, either store raw HTML (nvarchar(max)) or serialize rows to JSON and store that.
  • For performance use transactions, SqlBulkCopy column mappings, and batch inserts.
  • Respect robots.txt/terms and throttle requests; set a sensible User-Agent.

Useful parser: HtmlAgilityPack. For HttpClient and SqlBulkCopy usage see Microsoft docs.

Recommended Answers

All 5 Replies

Member Avatar for Mohtshm

u can do this as
following code should be used in codebehind page load method or any u should know better from where u want the desired out put should be saved.
<code> System.Text.StringBuilder sb = new StringBuilder();
TextWriter tWriter=new StringWriter(sb);
HtmlTextWriter htmlWriter=new HtmlTextWriter(tWriter);
this.Render(htmlWriter);
string outputResponseForDb=sb.ToString();
</code>

Member Avatar for khan17

sorry to ask u silly like this,

where should i place ur code..

help me pls

Member Avatar for Mohtshm

u should place in last line of page load method

Member Avatar for khan17

i want to fetch the table of contents. lets say if there is a web page which has the table i want to save that whole table contents on to sql server database.. how can i do it programmatically.

help me.

thanks

Member Avatar for Mohtshm

outputResponseForDB sttring has all html if there is only one table then use indexOf("table") to get the index of starting index and last index then substring on outputResponseForDB would do the job

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.