What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?

Principled construction of the quaternions

MaxCounters solution in C# from Codility

French license plates

How to find places to store/land a private airplane?

Is there a pattern for handling conflicting function parameters?

Are there types of animals that can't make the trip to space? (physiologically)

Missing quartile in boxplot

IEEE 754 square root with Newton-Raphson

What makes a character irredeemable?

Disable all sound permanently

麦酒 (ばくしゅ) for "beer"

Isn't the detector always measuring, and thus always collapsing the state?

How can Germany increase investments in Russia while EU economic sanctions against Russia are still in place?

Realistically, how much do you need to start investing?

Booting Ubuntu from USB drive on MSI motherboard -- EVERYTHING fails

Everyone Gets a Window Seat

Replace zeros in a list with last nonzero value

How dangerous is a very out-of-true disc brake wheel?

How to prove that the quadratic equation has exactly two real solutions

Would a horse be sufficient buffer to prevent injury when falling from a great height?

Does Bank Manager's discretion still exist in Mortgage Lending

When Vesuvan Shapeshifter copies turn face up replacement effects, why do they work?

Citing CPLEX 12.9

What is the score of my Scopa hand?



What is the idiomatic solution in SQL Server for reserving a block of ids for use in a bulk insert?


INSERT statements in a transaction and locking a range of rowsSequential GUID or bigint for 'huge' database table PKbulk insert for loas dataConfigure unconstrained delegation for BULK INSERTHow to handle errors in a transaction in a stored procedure?Sql server bulk insert remove text qualifierSQL Read Committed Snapshot Isolation in Database with Selects and Inserts OnlyLock Escalation happening while inserting few million records into a table in productionHow to prevent Deadlock on SELECT queries?






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty
margin-bottom:0;









3















I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.










share|improve this question





















  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    8 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    8 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    7 hours ago


















3















I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.










share|improve this question





















  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    8 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    8 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    7 hours ago














3












3








3








I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.










share|improve this question
















I have a table with an identity column and I want to reserve a block of ids which I can use for bulk inserting, whilst allowing inserts to still happen into that table.



Note this is part of a bulk insert of several tables, where those other tables relate to these ids via an FK. Therefore I need to block them out so I can prepare the relationships beforehand.



I've found a solution which works by taking a lock on the table in a transaction and then does the reseeding (which is pretty fast). But it looks a bit hacky to me - is there a generally accepted pattern for doing this?



create table dbo.test
(
id bigint not null primary key identity(1,1),
SomeColumn nvarchar(100) not null
)


Here's the code to block out (make room for) some ids:



declare @numRowsToMakeRoomFor int = 100

BEGIN TRANSACTION;

SELECT MAX(Id) FROM dbo.test WITH ( XLOCK, TABLOCK ) -- will exclusively lock the table whilst this tran is in progress,
--another instance of this query will not be able to pass this line until this instance commits

--get the next id in the block to reserve
DECLARE @firstId BIGINT = (SELECT IDENT_CURRENT( 'dbo.test' ) +1);

--calculate the block range
DECLARE @lastId BIGINT = @firstId + (@numRowsToMakeRoomFor -1);

--reseed the table
DBCC CHECKIDENT ('dbo.test',RESEED, @lastId);

COMMIT TRANSACTION;

select @firstId;


My code is batch processing blocks of data in chunks of about 1000. I have about a billion rows to insert in total. Everything is working fine - the database isn't the bottle neck, the batch processing itself is computationally expensive and requires me to add a couple of servers to run in parallel, so I need to accommodate more than one process "batch inserting" at the same time.







sql-server sql-server-2016 identity bulk-insert






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 6 hours ago









Paul White

59.8k16 gold badges310 silver badges489 bronze badges




59.8k16 gold badges310 silver badges489 bronze badges










asked 8 hours ago









Daniel James BryarsDaniel James Bryars

3741 gold badge3 silver badges16 bronze badges




3741 gold badge3 silver badges16 bronze badges










  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    8 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    8 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    7 hours ago













  • 2





    Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

    – ypercubeᵀᴹ
    8 hours ago












  • @ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

    – Daniel James Bryars
    8 hours ago






  • 2





    What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

    – ypercubeᵀᴹ
    7 hours ago








2




2





Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

– ypercubeᵀᴹ
8 hours ago






Why not use INSERT .. SELECT .. with OUTPUT? No locking, no reseeding. Just INSERT and get the IDs from OUTPUT so you can use them in the other tables.

– ypercubeᵀᴹ
8 hours ago














@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

– Daniel James Bryars
8 hours ago





@ypercubeᵀᴹ sorry, I see I didn't know you can Bulk insert with output. I'm doing the bulk insert from code (c#) which connects remotely - I guess I could write out files and then use bulk insert from a file. This might be a bit tricky.

– Daniel James Bryars
8 hours ago




2




2





What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

– ypercubeᵀᴹ
7 hours ago






What is the expensive, producing/calculating the rows? Or do you already have the data in files and just need to insert them? Also: how many tables and what are the relationships involved? Just 2 tables with an FK, many tables with a star schema, many tables with complex schema (cycles, multiple paths, etc)?

– ypercubeᵀᴹ
7 hours ago











1 Answer
1






active

oldest

votes


















5
















You can use procedure (introduced in SQL Server 2012):
sp_sequence_get_range



To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



There is an example:



CREATE SCHEMA Test ; 
GO

CREATE SEQUENCE Test.RangeSeq
AS int
START WITH 1
INCREMENT BY 1
CACHE 10
;

CREATE TABLE Test.ProcessEvents
(
EventID int PRIMARY KEY CLUSTERED
DEFAULT (NEXT VALUE FOR Test.RangeSeq),
EventTime datetime NOT NULL DEFAULT (getdate()),
EventCode nvarchar(5) NOT NULL,
Description nvarchar(300) NULL
) ;

DECLARE @range_first_value sql_variant ,
@range_first_value_output sql_variant ;

EXEC sp_sequence_get_range
@sequence_name = N'Test.RangeSeq'
, @range_size = 4
, @range_first_value = @range_first_value_output OUTPUT ;


Documentation: sp_sequence_get_range






share|improve this answer




























    Your Answer








    StackExchange.ready(function()
    var channelOptions =
    tags: "".split(" "),
    id: "182"
    ;
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function()
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled)
    StackExchange.using("snippets", function()
    createEditor();
    );

    else
    createEditor();

    );

    function createEditor()
    StackExchange.prepareEditor(
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader:
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/4.0/"u003ecc by-sa 4.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    ,
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    );



    );














    draft saved

    draft discarded
















    StackExchange.ready(
    function ()
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');

    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    5
















    You can use procedure (introduced in SQL Server 2012):
    sp_sequence_get_range



    To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



    There is an example:



    CREATE SCHEMA Test ; 
    GO

    CREATE SEQUENCE Test.RangeSeq
    AS int
    START WITH 1
    INCREMENT BY 1
    CACHE 10
    ;

    CREATE TABLE Test.ProcessEvents
    (
    EventID int PRIMARY KEY CLUSTERED
    DEFAULT (NEXT VALUE FOR Test.RangeSeq),
    EventTime datetime NOT NULL DEFAULT (getdate()),
    EventCode nvarchar(5) NOT NULL,
    Description nvarchar(300) NULL
    ) ;

    DECLARE @range_first_value sql_variant ,
    @range_first_value_output sql_variant ;

    EXEC sp_sequence_get_range
    @sequence_name = N'Test.RangeSeq'
    , @range_size = 4
    , @range_first_value = @range_first_value_output OUTPUT ;


    Documentation: sp_sequence_get_range






    share|improve this answer































      5
















      You can use procedure (introduced in SQL Server 2012):
      sp_sequence_get_range



      To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



      There is an example:



      CREATE SCHEMA Test ; 
      GO

      CREATE SEQUENCE Test.RangeSeq
      AS int
      START WITH 1
      INCREMENT BY 1
      CACHE 10
      ;

      CREATE TABLE Test.ProcessEvents
      (
      EventID int PRIMARY KEY CLUSTERED
      DEFAULT (NEXT VALUE FOR Test.RangeSeq),
      EventTime datetime NOT NULL DEFAULT (getdate()),
      EventCode nvarchar(5) NOT NULL,
      Description nvarchar(300) NULL
      ) ;

      DECLARE @range_first_value sql_variant ,
      @range_first_value_output sql_variant ;

      EXEC sp_sequence_get_range
      @sequence_name = N'Test.RangeSeq'
      , @range_size = 4
      , @range_first_value = @range_first_value_output OUTPUT ;


      Documentation: sp_sequence_get_range






      share|improve this answer





























        5














        5










        5









        You can use procedure (introduced in SQL Server 2012):
        sp_sequence_get_range



        To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



        There is an example:



        CREATE SCHEMA Test ; 
        GO

        CREATE SEQUENCE Test.RangeSeq
        AS int
        START WITH 1
        INCREMENT BY 1
        CACHE 10
        ;

        CREATE TABLE Test.ProcessEvents
        (
        EventID int PRIMARY KEY CLUSTERED
        DEFAULT (NEXT VALUE FOR Test.RangeSeq),
        EventTime datetime NOT NULL DEFAULT (getdate()),
        EventCode nvarchar(5) NOT NULL,
        Description nvarchar(300) NULL
        ) ;

        DECLARE @range_first_value sql_variant ,
        @range_first_value_output sql_variant ;

        EXEC sp_sequence_get_range
        @sequence_name = N'Test.RangeSeq'
        , @range_size = 4
        , @range_first_value = @range_first_value_output OUTPUT ;


        Documentation: sp_sequence_get_range






        share|improve this answer















        You can use procedure (introduced in SQL Server 2012):
        sp_sequence_get_range



        To use it you need to create a SEQUENCE object and use it as a default value instead of IDENTITY column.



        There is an example:



        CREATE SCHEMA Test ; 
        GO

        CREATE SEQUENCE Test.RangeSeq
        AS int
        START WITH 1
        INCREMENT BY 1
        CACHE 10
        ;

        CREATE TABLE Test.ProcessEvents
        (
        EventID int PRIMARY KEY CLUSTERED
        DEFAULT (NEXT VALUE FOR Test.RangeSeq),
        EventTime datetime NOT NULL DEFAULT (getdate()),
        EventCode nvarchar(5) NOT NULL,
        Description nvarchar(300) NULL
        ) ;

        DECLARE @range_first_value sql_variant ,
        @range_first_value_output sql_variant ;

        EXEC sp_sequence_get_range
        @sequence_name = N'Test.RangeSeq'
        , @range_size = 4
        , @range_first_value = @range_first_value_output OUTPUT ;


        Documentation: sp_sequence_get_range







        share|improve this answer














        share|improve this answer



        share|improve this answer








        edited 6 hours ago

























        answered 7 hours ago









        PiotrPiotr

        3989 bronze badges




        3989 bronze badges































            draft saved

            draft discarded















































            Thanks for contributing an answer to Database Administrators Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid


            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.

            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function ()
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdba.stackexchange.com%2fquestions%2f249625%2fwhat-is-the-idiomatic-solution-in-sql-server-for-reserving-a-block-of-ids-for-us%23new-answer', 'question_page');

            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            ParseJSON using SSJSUsing AMPscript with SSJS ActivitiesHow to resubscribe a user in Marketing cloud using SSJS?Pulling Subscriber Status from Lists using SSJSRetrieving Emails using SSJSProblem in updating DE using SSJSUsing SSJS to send single email in Marketing CloudError adding EmailSendDefinition using SSJS

            Кампала Садржај Географија Географија Историја Становништво Привреда Партнерски градови Референце Спољашње везе Мени за навигацију0°11′ СГШ; 32°20′ ИГД / 0.18° СГШ; 32.34° ИГД / 0.18; 32.340°11′ СГШ; 32°20′ ИГД / 0.18° СГШ; 32.34° ИГД / 0.18; 32.34МедијиПодациЗванични веб-сајту

            19. јануар Садржај Догађаји Рођења Смрти Празници и дани сећања Види још Референце Мени за навигацијуу