Showing posts with label TSQL. Show all posts
Showing posts with label TSQL. Show all posts

Friday, 3 June 2016

TSQL Snippet: Split string in records

Case
I have a string in TSQL and I want to split it into separate values / records. How do I do that?

Solution
There are a lot of split examples available on the web, but I really like the XQuery solution for this. First you add a begin XML tag in front of your list and an closing XML tag at the end. Then you replace all separators by a closing and a begin tag. After that you have an XML string and you can use Xquery to split it. Below a little snippet as part of a stored procedure, but you could also create a function for it or just use the three lines in your own code:

-- Snippet
CREATE PROCEDURE [dbo].[SplitList] (
      @List VARCHAR(255)
    , @Separator VARCHAR(1)
)
as
BEGIN
    DECLARE @Split XML;
    SET @Split = CAST('<t>' + REPLACE(@List, @Separator, '</t><t>') + '</t>' as XML) 
    SELECT Col.value('.', 'VARCHAR(255)') as ListValue FROM @Split.nodes('t') as xmlData(Col)  order by 1
END


Note: your string / list can't contain forbidden XML characters like <, > and &. You could use additional REPLACE functions to prevent errors: REPLACE(@List,"<", "&lt;")
split snippet

Wednesday, 28 January 2015

Insert unknown dimension record for all dimension tables

Case
I have a lot of dimension packages in SSIS that all insert a default record for unknown dimension values. It's a lot of repetitive and boring work. Is there an alternative for creating an insert query manually?
A typical dimension package





















Solution
Instead of creating an insert query manually for each dimension table you could also create a Stored Procedure to do this for you. Instead of the insert query in the Execute SQL Task you execute this Stored Procedure in the Execute SQL Task.
-- TSQL code
USE [datamart]
GO

/****** datamart:  StoredProcedure [dbo].[InsertUnknownDimensionRow]   ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE PROCEDURE [dbo].[InsertUnknownDimensionRow](@TableName nvarchar(128))
AS
BEGIN

 -- This Stored Procedure inserts a record in the dimension table
 -- for unknown dimension values. It generates an insert statement
 -- based on the column datatypes and executes it.
 -- The integer column with identity enabled gets the value -1 and
 -- all other columns get a default value based on their datatype.
 -- Columns with a default value are ignored.

 -- Create temporary table for column specs of dimension table
 DECLARE @TableSpecs TABLE (
        COLUMN_ID     int identity,
        COLUMN_NAME     nvarchar(128),
        DATA_TYPE     nvarchar(128),
        CHARACTER_MAXIMUM_LENGTH int,
        COLUMN_IS_IDENTITY   bit
          )

 -- Use the information schema to get column info and insert it
 -- to the temporary table.
 INSERT              @TableSpecs
 SELECT              C.COLUMN_NAME
 ,                   C.DATA_TYPE
 ,                   C.CHARACTER_MAXIMUM_LENGTH
 ,                   columnproperty(object_id(C.TABLE_SCHEMA + '.' + C.TABLE_NAME)
      , C.COLUMN_NAME, 'IsIdentity') AS COLUMN_IS_IDENTITY
 FROM                INFORMATION_SCHEMA.COLUMNS C
 WHERE               QUOTENAME(C.TABLE_NAME) = QUOTENAME(@TableName)
 AND                 C.COLUMN_DEFAULT IS NULL
 ORDER BY            C.ORDINAL_POSITION

 -- Variables to keep track of the number of columns
 DECLARE @ColumnId INT
 SET @ColumnId = -1

 DECLARE @ColumnCount INT
 SET @ColumnCount = 0

 -- Variables to create the insert query
 DECLARE @INSERTSTATEMENT_START nvarchar(max)
 DECLARE @INSERTSTATEMENT_END nvarchar(max)

 SET @INSERTSTATEMENT_START = 'INSERT INTO ' + QUOTENAME(@TableName) + ' ('
 SET @INSERTSTATEMENT_END = 'VALUES ('

 -- Variables to complete the insert query with
 -- extra enable and disable identity statements
 -- You could add an extra check in the loop to
 -- make sure there is an identity column in the
 -- table. Otherwise the SET IDENTITY_INSERT
 -- statement will fail.
 DECLARE @IDENITYSTATEMENT_ON nvarchar(255)
 DECLARE @IDENITYSTATEMENT_OFF nvarchar(255)

 SET @IDENITYSTATEMENT_ON = 'SET IDENTITY_INSERT ' + QUOTENAME(@TableName) + ' ON;'
 SET @IDENITYSTATEMENT_OFF = 'SET IDENTITY_INSERT ' + QUOTENAME(@TableName) + ' OFF;'

 -- Variables filled and use the WHILE loop
 DECLARE @COLUMN_NAME VARCHAR(50)
 DECLARE @DATA_TYPE VARCHAR(50)
 DECLARE @CHARACTER_MAXIMUM_LENGTH INT
 DECLARE @COLUMN_IS_IDENTITY BIT

 -- WHILE loop to loop through all columns and
 -- create a insert query with the columns
 WHILE @ColumnId IS NOT NULL
 BEGIN
   -- Keep track of the number of columns
   SELECT @ColumnId = MIN(COLUMN_ID)
   ,       @ColumnCount = @ColumnCount + 1
   FROM    @TableSpecs
   WHERE   COLUMN_ID > @ColumnCount

   -- Check if there are any columns left
   IF @ColumnId IS NULL
   BEGIN
    -- No columns left, break loop
    BREAK
   END
   ELSE
   BEGIN
    -- Get info for column number x
    SELECT       @COLUMN_NAME = COLUMN_NAME
    ,            @DATA_TYPE = DATA_TYPE
    ,            @CHARACTER_MAXIMUM_LENGTH = CHARACTER_MAXIMUM_LENGTH
    ,            @COLUMN_IS_IDENTITY = COLUMN_IS_IDENTITY
    FROM         @TableSpecs
    WHERE        COLUMN_ID = @ColumnCount
   END
       
   -- Start building the begin of the statement (same for each column)
   SET @INSERTSTATEMENT_START = @INSERTSTATEMENT_START + @COLUMN_NAME + ','

   -- Start building the end of the statement (the default values)
   IF @COLUMN_IS_IDENTITY = 1
   BEGIN
    -- Default value if the current column is the identity column
    SET @INSERTSTATEMENT_END = @INSERTSTATEMENT_END + '-1,'
   END
             
   IF @DATA_TYPE IN ('int', 'numeric', 'decimal', 'money', 'float', 'real', 'bigint', 'smallint', 'tinyint', 'smallmoney') AND (@COLUMN_IS_IDENTITY = 0)
   BEGIN
    -- Default value if the current column is a numeric column,
    -- but not an identity: zero
    SET @INSERTSTATEMENT_END = @INSERTSTATEMENT_END + '0,'
   END

   IF @DATA_TYPE IN ('char', 'nchar', 'varchar', 'nvarchar')
   BEGIN
    -- Default value if the current column is a text column
    -- Part of the text "unknown" depending on the length
    SET @INSERTSTATEMENT_END = @INSERTSTATEMENT_END + '''' + LEFT('Unknown', @CHARACTER_MAXIMUM_LENGTH) + ''','
   END

   IF @DATA_TYPE IN ('datetime', 'date', 'timestamp', 'datatime2', 'datetimeoffset', 'smalldatetime', 'time') 
   BEGIN
    -- Default value if the current column is a datetime column
    -- First of january 1900
    SET @INSERTSTATEMENT_END = @INSERTSTATEMENT_END + '''' + CONVERT(varchar, CONVERT(date, 'Jan 1 1900')) + ''','
   END

   IF @DATA_TYPE = 'bit' 
   BEGIN
    -- Default value if the current column is a boolean 
    SET @INSERTSTATEMENT_END = @INSERTSTATEMENT_END + '0,'
   END
 END

 -- Remove last comma from start and end part of the insert statement
 SET @INSERTSTATEMENT_START = LEFT(@INSERTSTATEMENT_START, LEN(@INSERTSTATEMENT_START) - 1) + ')'
 SET @INSERTSTATEMENT_END = LEFT(@INSERTSTATEMENT_END, LEN(@INSERTSTATEMENT_END) - 1) + ');'

 -- Execute the complete statement
 EXEC (@IDENITYSTATEMENT_ON + ' ' + @INSERTSTATEMENT_START + ' ' + @INSERTSTATEMENT_END + ' ' + @IDENITYSTATEMENT_OFF)
      
END

GO
-- Tweak the code for your own needs and standards
-- Optional extra check if you don't want to truncate
-- your dimensions: is there already a default/unknown
-- record available

Execute Stored Procedure


















Note: only the most common datatypes are handled. Add more if-statements if you expect data types like varbinary, xml, image or sql_variant

Saturday, 24 May 2014

Create and fill Age dimension

Case
Is there an easy way to create and populate an age dimension with age groups?

Solution
Creating an age dimension is usually done once and probably not in SSIS, but with a TSQL script.
For each new assignment I use a script similar to this and adjust it to the requirements for that particular assignment.

-- Drop dimension table if exists
IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[dim_age]') AND TYPE IN (N'U'))
BEGIN
 DROP TABLE [dbo].[dim_age]
END

-- Create table dim_age
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

CREATE TABLE [dbo].[dim_age](
  [dim_age_id] [int] IDENTITY(-1,1) NOT NULL,
  [Age] [smallint] NULL,
  [AgeGroup1] [nvarchar](50) NULL,
  [AgeGroup1Sort] [int] NULL,
  [AgeGroup2] [nvarchar](50) NULL,
  [AgeGroup2Sort] [int] NULL,
 CONSTRAINT [PK_dim_age] PRIMARY KEY CLUSTERED 
(
 [dim_age_id] ASC
) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
GO

-- Enter unknown dimension value (in case a person's date of birth is unknown)
INSERT INTO [dbo].[dim_age]
           ([Age]
           ,[AgeGroup1]
     ,[AgeGroup1Sort]
           ,[AgeGroup2]
     ,[AgeGroup2Sort])
     VALUES
           (-1
           ,'Unknown'
     ,0
           ,'Unknown'
     ,0)
GO

-- Enter all ages
declare @age smallint;
set @age = 0;

-- Loop through ages 0 to 130
WHILE @age < 131
BEGIN
 INSERT INTO [dbo].[dim_age]
  ([Age]
  ,[AgeGroup1]
  ,[AgeGroup1Sort]
  ,[AgeGroup2]
  ,[AgeGroup2Sort])
 VALUES
  (@age
  -- Use the common age groups/categories of your region/branch/industry
  -- This is just an example
  , CASE
    WHEN @age < 15 THEN '0 till 15 year'
    WHEN @age < 25 THEN '15 till 25 year'
    WHEN @age < 35 THEN '25 till 35 year'
    WHEN @age < 45 THEN '35 till 45 year'
    WHEN @age < 55 THEN '45 till 55 year'
    WHEN @age < 65 THEN '55 till 65 year'
    ELSE '65 year and older'
   END
  -- Add value to sort on in SSAS
  , CASE
    WHEN @age < 15 THEN 1
    WHEN @age < 25 THEN 2
    WHEN @age < 35 THEN 3
    WHEN @age < 45 THEN 4
    WHEN @age < 55 THEN 5
    WHEN @age < 65 THEN 6
    ELSE 7
   END
  , CASE
    WHEN @age < 19 THEN 'Juvenile'
    ELSE 'Mature'
   END
  -- Add value to sort on in SSAS
  , CASE
    WHEN @age < 19 THEN 1
    ELSE 2
   END
  )

 -- Goto next age
 set @age = @age + 1
END

The result: filled age dimension
























How could you use this dimension?
A while ago I also posted an example to create and populate a date dimension. So now you can combine those in a datamart. I have an employee table and an absence table with a start- and enddate.
Employee table

Absence table



















I will use the date dimension to split the absence time periods in separate days and then calculate the employee's age of each day of absence. This will go in to a fact table and then I can use the age dimension to see absence per age group.
-- Split absence time periode in separate days, but go back 2 years max and 1 year forward if end date is unknown
SELECT  Absence.AbsenceId
,   Absence.EmployeeNumber
--   Date of absence
,   dim_date.Date as AbsenceDate
,   Absence.ReasonCode
--   Calculation of age at time of absence
,   DATEDIFF(YEAR, Employee.DateOfBirth, dim_date.Date)
   -
   (CASE
    WHEN DATEADD(YY, DATEDIFF(YEAR, Employee.DateOfBirth, dim_date.Date), Employee.DateOfBirth)
     >  dim_date.Date THEN 1
    ELSE 0
   END) as Age
FROM  EmployeeApplication.dbo.Absence
INNER JOIN EmployeeApplication.dbo.Employee
   on Absence.EmployeeNumber = Employee.EmployeeNumber
INNER JOIN  DM_Staff.dbo.dim_date
   on dim_date.Date
   -- change start date to lower bound if it's below it
            BETWEEN CASE WHEN YEAR(Absence.AbsenceStartDate) >= YEAR(GETDATE()) - 2 THEN Absence.AbsenceStartDate
            ELSE DATEADD(yy, DATEDIFF(yy, 0, getdate()) - 2, 0) END
   -- change end date to upper bound if it's null
            AND ISNULL(Absence.AbsenceEndDate, DATEADD(yy, DATEDIFF(yy, 0, getdate()) + 2, -1))
--   Filter absence record with an enddate below the lower bound (perhaps a bit superfluous with the inner join)
WHERE  YEAR(ISNULL(Absence.AbsenceEndDate, GETDATE())) >= YEAR(GETDATE()) - 2


Result of query that can be used in a fact package





















fact absence


























Note: this is a simplified situation to keep things easy to explain.

Sunday, 20 January 2013

SSIS 2012 Data taps

Case
I have a (very basic) package and want to add data taps to it in the Integration Services Catalogs.
My package adding colors to a table






















Solution
Datataps are the 'dataviewers' for packages within the Integration Services Catalogs, but their output is to a file instead of to the screen. They can be added with the stored procedures from the SSISDB.
My package in the SSIS Catalog

















1) Execution
We first have to create an execution and we need its execution id in the next stored procedure calls. The Folder, Project and Packagename can be found in the picture above.
-- Create a variable to store the ID of the package execution
DECLARE @execution_id bigint

-- Create a package execution and fill the variable
EXECUTE [SSISDB].[catalog].[create_execution] 
  @folder_name = 'ilionx'
  ,@project_name = 'DataTap'
  ,@package_name = 'DimColors.dtsx' 
  ,@reference_id = null
  ,@use32bitruntime = false
  ,@execution_id = @execution_id OUTPUT

-- Add some optional parameters like Verbose logging
EXECUTE [SSISDB].[catalog].[set_execution_parameter_value]
  @execution_id = @execution_id
  ,@object_type=50
  ,@parameter_name=N'LOGGING_LEVEL'
  ,@parameter_value=3 -- Verbose


2) Data taps
Now we have an execution, we can add data taps to it. For this we need to know the PackagePath or GUID of the Data Flow Task you want to tap.

The (GU)ID and PackagePath of the Data Flow Task.


























And we need to know the IdentificationString of the Data Flow Path within the Data Flow Task.
Data Flow Path properties






















I will add one data tap with the PackagePath and the other on the next Data Flow Path with the GUID.
-- Create a data type with the data flow PackagePath 
EXECUTE [SSISDB].[catalog].[add_data_tap]
  @execution_id = @execution_id
  ,@task_package_path = '\Package\Add Colors'
  ,@dataflow_path_id_string = 'Paths[SRC - Colors.Flat File Source Output]'
  ,@data_filename = 'ssisjoost1.txt'
  ,@max_rows = 10

-- Create a data type with the data flow ID
EXECUTE [SSISDB].[catalog].add_data_tap_by_guid
  @execution_id = @execution_id
  ,@dataflow_task_guid = '{9DE67956-E158-4264-AA98-F9C07A7C7731}'
  ,@dataflow_path_id_string = 'Paths[DER - Uppercase.Derived Column Output]'
  ,@data_filename = 'ssisjoost2.txt'
  ,@max_rows = 10


3) Execute
The last step is to execute the created execution and watch the output folder (C:\Program Files\Microsoft SQL Server\110\DTS\DataDumps) for new files.
-- Execute the created execution
EXECUTE [SSISDB].[catalog].[start_execution]
  @execution_id = @execution_id





















Note: For steps 1 and 3 of above you can also use the Script button. Then you only have to add the code from step 2. See steps in this picture:

Friday, 3 August 2012

SSMS: Prevent saving changes that require table re-creation

Case
When changing a column in a table I get this message preventing me to save the changes:
Saving changes is not permitted. The changes you have made require the
following tables to be dropped and recreated. You have either made
changes to a table that can't be re-created or enabled the option
Prevent saving changes that require the table to be re-created.


























This question got nothing to do with SSIS self, but editing tables is a common task for SSIS developers. And I'm always browsing a couple of minutes to find the right option to disable.


Solution
1) Menu Tools
Go to the Tools menu and select "Options..."
Tools


















2) Options, Designers
Go to Designers and disable Prevent saving changes that require table re-creation.
Designers

Thursday, 1 September 2011

SSIS Transactions with TSQL

Case
A couple of months ago I did a post on Transactions in SSIS, but that solution requires enabling the windows service Microsoft Distributed Transaction Coordinator (MS DTC). What if you can't (or prefer not to) use that service?

Solution
You can use the Transact SQL transactions to accomplish the same result. Same example as before. I want to empty and refill a table with values from a CSV file, but I want to keep the old data when the refill fails. My package:
Example



















1) Container
Add a Sequence Container and drag the existing Execute SQL Task (which empties the table) and the Data Flow Task (which fills the table) to it.
Sequence Container



















2) Start Transaction
Add an Execute SQL Task before the Sequence Container and use the same connection as in the other tasks. Enter the following statement in the SQLStatement field: BEGIN TRAN MyTran.
Start Transaction






















BEGIN TRAN



















3) Commit Transaction
Add an Execute SQL Task after the Sequence Container and use the same connection as in the other tasks. Enter the following statement in the SQLStatement field: COMMIT TRAN MyTran.
Commit Transaction






















COMMIT TRAN



















4) Rollback Transaction
Add an other Execute SQL Task after the Sequence Container and use the same connection as in the other tasks. Enter the following statement in the SQLStatement field: ROLLBACK TRAN MyTran.
Rollback Transaction






















ROLLBACK TRAN



















5) Precedence Contraint
Change the Value property of the Precedence Contraint between the Sequence Container and the Rollback from Success to Failure. If something fails in the Sequence Container the Rollback command will be executed.
Precedence Contraint






















Failure

















6) RetainSameConnection
Now the most important thing. Change the RetainSameConnection property of the database connection from false to True.
RetainSameConnection






















7) The Result
That's all there is. Now you can test your package. You can open the CSV file in Excel to lock the file and fail the package.
The Result
















* UPDATE *
Added the optional transactionname in case you want to re-execute. See comment Arthur Zubarev.

Friday, 4 February 2011

Create a GUID column in SSIS

Case
How do you create a new Guid column in SSIS?

Solution
There is no SSIS function for that, but there are a few workarounds.

A) If you already have a valid GUID in your source, but it's still a string type, then you can use a Derived Column to create a real guid.
(DT_GUID) ("{" + YourGuid + "}")



















B) If your source is a SQL Server table, you can use the TSQL statement to generate a Guid column.
NEWID() as Guid


















C) Or you can use a Script Component to generate a new Guid Column:

1) Script Component
Add a Script Component in your Data Flow and select Transformation as the Script Component Type.
Transformation Type













2) Create new column
Edit the Script Component and goto the Inputs and Outputs tab. Expand the Output 0 and add a new column. The column type should be unique identifier [DT_GUID].
Add new Guid column



















3) The Script
Edit the Script. Remove the PreExecute and PostExecute methods and add the following code to the Input0_ProcessInputRow method. That's all. Only one row of code.
// C# code
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;

[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
    public override void Input0_ProcessInputRow(Input0Buffer Row)
    {
        // Create a Globally Unique Identifier with SSIS
        Row.Guid = System.Guid.NewGuid(); 
    }
}

' VB.net code
Imports System
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Pipeline.Wrapper
Imports Microsoft.SqlServer.Dts.Runtime.Wrapper

<microsoft.sqlserver.dts.pipeline.ssisscriptcomponententrypointattribute> _
<clscompliant(false)> _
Public Class ScriptMain
    Inherits UserComponent

    Public Overrides Sub Input0_ProcessInputRow(ByVal Row As Input0Buffer)
        ' Create a Globally Unique Identifier with SSIS
        Row.Guid = System.Guid.NewGuid()
    End Sub
End Class

Thursday, 13 January 2011

Create and fill Time dimension

Case
Is there an easy way to create and populate a time(/date) dimension?

Solution
Creating a time dimension is usually done once and probably not in SSIS, but with a TSQL script.
For each new assignment I use this script and adjust it to the requirements for that particular assignment .
-- Delete time dimension if it already exists.
IF Exists(Select Name from sysobjects where name = 'Dim_Time')
BEGIN
    Drop Table Dim_Time
END
GO

-- Standard options for creating tables
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

-- Create your dimension table
-- Adjust to your own needs
Create Table dbo.Dim_Time
(
    Dateid int IDENTITY (1,1) PRIMARY KEY CLUSTERED,
    Date date,
    DateString varchar(10),
    Day int,
    DayofYear int,
    DayofWeek int,
    DayofWeekName varchar(10),
    Week int,
    Month int,
    MonthName varchar(10),
    Quarter int,
    Year int,
    IsWeekend bit,
    IsLeapYear bit
)

-- Declare and set variables for loop
Declare
@StartDate datetime,
@EndDate datetime,
@Date datetime

Set @StartDate = '2000/01/01'
Set @EndDate = '2020/12/31'
Set @Date = @StartDate

-- Loop through dates
WHILE @Date <=@EndDate
BEGIN
    -- Check for leap year
    DECLARE @IsLeapYear BIT
    IF ((Year(@Date) % 4 = 0) AND (Year(@Date) % 100 != 0 OR Year(@Date) % 400 = 0))
    BEGIN
        SELECT @IsLeapYear = 1
    END
    ELSE
    BEGIN
        SELECT @IsLeapYear = 0
    END

    -- Check for weekend
    DECLARE @IsWeekend BIT
    IF (DATEPART(dw, @Date) = 1 OR DATEPART(dw, @Date) = 7)
    BEGIN
        SELECT @IsWeekend = 1
    END
    ELSE
    BEGIN
        SELECT @IsWeekend = 0
    END

    -- Insert record in dimension table
    INSERT Into Dim_Time
    (
    [Date],
    [DateString],
    [Day],
    [DayofYear],
    [DayofWeek],
    [Dayofweekname],
    [Week],
    [Month],
    [MonthName],
    [Quarter],
    [Year],
    [IsWeekend],
    [IsLeapYear]
    )
    Values
    (
    @Date,
    CONVERT(varchar(10), @Date, 105), -- See links for 105 explanation
    Day(@Date),
    DATEPART(dy, @Date),
    DATEPART(dw, @Date),
    DATENAME(dw, @Date),
    DATEPART(wk, @Date),
    DATEPART(mm, @Date),
    DATENAME(mm, @Date),
    DATENAME(qq, @Date),
    Year(@Date),
    @IsWeekend,
    @IsLeapYear
    )

    -- Goto next day
    Set @Date = @Date + 1
END
GO

Interesting links:
CAST and CONVERT:http://msdn.microsoft.com/en-us/library/ms187928.aspx
DATEPARThttp://msdn.microsoft.com/en-us/library/ms174420.aspx
DATENAMEhttp://msdn.microsoft.com/en-us/library/ms174395.aspx

Let me know if you have an interesting addition for this script that could help others.
Related Posts Plugin for WordPress, Blogger...