hello i am working on this application where i need to import 60k+ user data and insert into different table. i have made a C# application which selects the database and inserts from a text file.like

insert into dbo.dbname ([user_name],[u_id]) values (a,b)

but this apps hangs for 60K+ data so i need to use bulk insert
All this table has two field common namely user_name and u_id so my text input file contains just this two values user_name and u_id(other column values are dynamically inserted like guid,systimestamp..etc which varies for differnt table).

can i have a bulk insert into specific columns ..for example just in user_name and u_id?

the sample of my input text file is given below

abcdedf cd123
hohower ad120
bubble ak324
messi ar016

eagerly waiting to hear from you all
P.S i am using sql server 2005

Dani AI

Generated

Good points from about format files — they let a server-side bulk operation map or skip fields. The key reason your view trick failed is that BULK INSERT runs on the SQL Server host and expects the data file to be reachable from that host (local path or a UNC share that the SQL Server service can access). For a remote client that owns the file, don’t rely on server-side BULK INSERT unless you can place the file where the server can read it. (learn.microsoft.com)

For a C# client the simplest, fastest fix is SqlBulkCopy: it streams from your process (no server-side file required) and supports explicit column mappings so you can load only user_name and u_id and let the table defaults (GUID, GETDATE, etc.) populate the rest. Use ColumnMappings and set sensible BatchSize/BulkCopyTimeout to avoid long-running single transactions. If you need triggers or constraint checking during the load, enable the corresponding SqlBulkCopyOptions flags (FireTriggers, CheckConstraints). Example pattern:

// build a small DataTable with the two columns, or implement an IDataReader
using(var conn = new SqlConnection(connString))
using(var bulk = new SqlBulkCopy(conn))
{
    bulk.DestinationTableName = "dbo.YourTable";
    bulk.ColumnMappings.Add("user_name","user_name");
    bulk.ColumnMappings.Add("u_id","u_id");
    bulk.BatchSize = 5000;
    conn.Open();
    bulk.WriteToServer(dataTable);
}

SqlBulkCopy docs and option flags explain the properties and flags. (learn.microsoft.com)

If you prefer a command-line route, run the bcp utility from the client machine (it sends the file to SQL Server over the network and can use a format file to map/skip fields). Example:

bcp "MyDb.dbo.YourTable" in "C:\temp\users.txt" -S MyServer -T -c -t " " -f "C:\temp\users.fmt"

Use bcp when you want a quick file->table tool; use BULK INSERT only when the file can be placed on a path the server can read. Note: BULK INSERT always enforces UNIQUE/PK constraints and will error on duplicates; CHECK/FK behavior differs unless you request constraint checking. (learn.microsoft.com)

Practical checklist: stream the input (avoid building huge strings), use SqlBulkCopy with column mappings and a moderate BatchSize, load into a staging table and MERGE into production if you need de-duplication or validation, and re‑validate constraints after a bulk load. For ~60K rows SqlBulkCopy with BatchSize set should complete quickly and not hang.

Recommended Answers

All 2 Replies

can i have a bulk insert into specific columns ..for example just in user_name and u_id?

yes, you need a format file when inserting data in a column's subset. In format file you specify which columns are to insert and which are to overleap.

You may also read there where some aspects of bulk insert had been discussed.

-- tesu

yeah i came with another technique ...i just created a view of my table with just required attribute column and inserted the data on the view...but it wasnt quite useful.
bulk insert expects a local input file and what i am doing is through a remote client ..
do you think there is any other way to do that and just reading through the links which u gave me ..seems primary key constraints is not maintained.

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.