What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?
Principled construction of the quaternions
MaxCounters solution in C# from Codility
French license plates
How to find places to store/land a private airplane?
Is there a pattern for handling conflicting function parameters?
Are there types of animals that can't make the trip to space? (physiologically)
Missing quartile in boxplot
IEEE 754 square root with Newton-Raphson
What makes a character irredeemable?
Disable all sound permanently
麦酒 (ばくしゅ) for "beer"
Isn't the detector always measuring, and thus always collapsing the state?
How can Germany increase investments in Russia while EU economic sanctions against Russia are still in place?
Realistically, how much do you need to start investing?
Booting Ubuntu from USB drive on MSI motherboard -- EVERYTHING fails
Everyone Gets a Window Seat
Replace zeros in a list with last nonzero value
How dangerous is a very out-of-true disc brake wheel?
How to prove that the quadratic equation has exactly two real solutions
Would a horse be sufficient buffer to prevent injury when falling from a great height?
Does Bank Manager's discretion still exist in Mortgage Lending
When Vesuvan Shapeshifter copies turn face up replacement effects, why do they work?
Citing CPLEX 12.9
What is the score of my Scopa hand?
What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?
INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
add a comment
|
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
2
Why not useINSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.
– ypercubeᵀᴹ
8 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
8 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
7 hours ago
add a comment
|
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.
Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.
I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?
create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)
Here's the code to block out (make room for) some ids:
declare @numRowsToMakeRoomFor int = 100
BEGIN TRANSACTION;
SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits
--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);
--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);
--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);
COMMIT TRANSACTION;
select @firstId;
My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.
sql-server sql-server-2016 identity bulk-insert
sql-server sql-server-2016 identity bulk-insert
edited 6 hours ago
Paul White♦
59.8k16 gold badges310 silver badges489 bronze badges
59.8k16 gold badges310 silver badges489 bronze badges
asked 8 hours ago
Daniel James BryarsDaniel James Bryars
3741 gold badge3 silver badges16 bronze badges
3741 gold badge3 silver badges16 bronze badges
2
Why not useINSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.
– ypercubeᵀᴹ
8 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
8 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
7 hours ago
add a comment
|
2
Why not useINSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.
– ypercubeᵀᴹ
8 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
8 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
7 hours ago
2
2
Why not use
INSERT .. SELECT ..
with OUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.– ypercubeᵀᴹ
8 hours ago
Why not use
INSERT .. SELECT ..
with OUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.– ypercubeᵀᴹ
8 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
8 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
8 hours ago
2
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
7 hours ago
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
7 hours ago
add a comment
|
1 Answer
1
active
oldest
votes
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
add a comment
|
Your Answer
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "182"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
add a comment
|
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
add a comment
|
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
You can use procedure (introduced in SQL Server 2012):sp_sequence_get_range
To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.
There is an example:
CREATE SCHEMA Test ;
GO
CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;
CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;
DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;
EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;
Documentation: sp_sequence_get_range
edited 6 hours ago
answered 7 hours ago
PiotrPiotr
3989 bronze badges
3989 bronze badges
add a comment
|
add a comment
|
Thanks for contributing an answer to Database Administrators Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
2
Why not use
INSERT .. SELECT ..
withOUTPUT
? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.– ypercubeᵀᴹ
8 hours ago
@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.
– Daniel James Bryars
8 hours ago
2
What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?
– ypercubeᵀᴹ
7 hours ago