Showing posts with label SQL. Show all posts
Showing posts with label SQL. Show all posts

Thursday, 14 October 2010

SQL: Converting Column Datatype from DateTime to Uniqueidentifier

Another day for weird annoying things.

I had a table on which I'd inadvertantly set a columns datatype to datetime when it should have been uniqueidentifier. The table was empty.

Resetting the datatype via script won't work due as SQL knows it can't implicitly convert the (nonexistant) data. So I tried using the GUI to change the datatype and I got incompatible operand errors. Tried doing an ALTER TABLE command and got the same error. Finally I tricked the little bugger by converting it to varchar first then to uniqueidentifier. Boom! Done. Sorted.

SSIS Doesn't Play Nice with Special Characters

Oh SSIS you cantankerous old thing, you! I spent an hour today adding cleansing tasks into several data flows to replace special characters with other values ie "" to "-" and "[" to "(" etc. Only to have the content of those dataflows disappear when I closed then reopened the saved package. When I say disappear, I mean empty. Completely. Devoid of all steps. An empty shell.

Only dataflows which had the new special character cleansing step were empty - the rest were fine. This placed the blame squarely on the new steps. I should have twigged when I attempted to copy the step from one dataflow to another and got an error - something about HEX values being invalid. Stupid me, blindly continued creating them manually in each dataflow. Once done, I saved and closed. Then when I reopened, those dataflows were completely empty.

Fortunately being a concientous backer-upper-er-er, I had the previous version stored elsewhere. Sans special character cleansing of course.

So first, why do I need to do cleansing at all? These characters have passed through SSIS into the staging database with no problems. They have also passed through SSIS into the datamart database with no hiccups. Well the problem comes when I process the dimensions which use the datamart tables as their source. Then I get errors.

So, for whatever reason, cleansing ahoy!
But SSIS won't do it, as proven by my frustrating waste of an hour.
I tried using them directly in my source query with the same result. So I created udf_CleanseSpecialCharacters to handle it.


/**********************************************************************************************************************************************
HEADER
/***********************************************************************************************************************************************
FN Name: dbo.udf_CleanseSpecialCharacters
Author: Kristen Hodges
Date: 14 October 2010

-------------------------------------------------------------------------------------------------------------------------------------------
Purpose:
--------------------------------------------------------------------------------------------------------------------------------------------
Return a given string, cleansed of special characters

--------------------------------------------------------------------------------------------------------------------------------------------
OVERVIEW
--------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------------------------------
NOTES
--------------------------------------------------------------------------------------------------------------------------------------------

--------------------------------------------------------------------------------------------------------------------------------------------
ASSUMPTIONS AND BUSINESS RULES
--------------------------------------------------------------------------------------------------------------------------------------------
It may be desirable in certain contexts to retain some characters whereas in others it is not
eg comma stored in table but not as a ssas dimension member unique key
The @iOptional_CharactersToCleanseList is an optional parameter which can be used to pass in a string of characters which should be stripped and
any others will be retained

The @iStripCharacters is a true/false flag which will strip the characters if true
or if false, the value will be replaced with particular values eg [ becomes -

--------------------------------------------------------------------------------------------------------------------------------------------
USAGE
--------------------------------------------------------------------------------------------------------------------------------------------
Called by:

Sample USage:
select dbo.udf_CleanseSpecialCharacters('John&kathy@somedomain.com.au',1,0,NULL)
select dbo.udf_CleanseSpecialCharacters('John&kathy@somedomain.com.au',1,1,NULL)
select dbo.udf_CleanseSpecialCharacters('John&kathy@somedomain.com.au',1,1,'&')

--------------------------------------------------------------------------------------------------------------------------------------------
CHANGE LOG
--------------------------------------------------------------------------------------------------------------------------------------------
AUTHOR DATE CHANGE
KHODGES 2010-10-14 Created

************************************************************************************************************************************************/

************************************************************************/

ALTER FUNCTION [dbo].[udf_CleanseSpecialCharacters](
@iStringToCleanse varchar(500)
,@iStripCharacters bit
,@iStripAllSpaces bit
,@iOptional_CharactersToCleanseList varchar(50)
)
RETURNS varchar(500) AS BEGIN

DECLARE @return varchar(500), @CurrentChar char(1), @CurrentReplacement varchar(10), @posn int

SET @return = @iStringToCleanse

set @CurrentChar = CHAR(5) --
if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = '-'
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(58) -- :
if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = '-'
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(59) -- ;
if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = '-'
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(91) -- [
if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = '-'
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(93) -- ]
set @CurrentReplacement = ''
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(37) -- %
set @CurrentReplacement = ''
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(38) -- &
if @iStripCharacters = 0 set @CurrentReplacement = '' else begin
set @CurrentReplacement = ' and '
IF @iStripAllSpaces = 1 set @CurrentReplacement = replace(@CurrentReplacement,' ','-')
end
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(124)-- |

if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = '-'
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(44) -- ,
if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = SPACE(1)
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(36) -- $
set @CurrentReplacement = ''
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(35) -- #
if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = 'No.'
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(35) -- #
if @iStripCharacters = 0 set @CurrentReplacement = '' else set @CurrentReplacement = 'No.'
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(64) -- @
if @iStripCharacters = 0 set @CurrentReplacement = '' else begin
set @CurrentReplacement = ' at '
IF @iStripAllSpaces = 1 set @CurrentReplacement = replace(@CurrentReplacement,' ','-')
end
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)

set @CurrentChar = CHAR(46) -- .
if @iStripCharacters = 0 set @CurrentReplacement = '' else begin
set @CurrentReplacement = ' dot '
IF @iStripAllSpaces = 1 set @CurrentReplacement = replace(@CurrentReplacement,' ','-')
end
IF PATINDEX('%'+@CurrentChar+'%',@iOptional_CharactersToCleanseList) >= 1 OR @iOptional_CharactersToCleanseList IS NULL
SET @return = REPLACE(@return,@CurrentChar,@CurrentReplacement)


IF @iStripAllSpaces = 1 SET @return = REPLACE(@return,' ','')

RETURN @return
END

Thursday, 20 May 2010

Visual Studio 2010 Trial: SQL Project Limitation: Refactor/Rename

Please, somebody correct me if I'm wrong...

** UPDATED **

Yep I'm an idiot. Somehow I installed Professional edition not Ultimate. And professional does not support schema compare or refactoring. Duh!


The Visual Studio 2010 trial claims to be functionally unlimited - just time-limited (90 days). BUT - from what I can tell, object name refactoring (renaming) is unavailable.

According to MSDN it can be done by "right-click the object in Schema view, point to Refactor, and then click Rename'.... or from the Data menu, point to Refactor etc...


However, there is no Refactor option in either location.



MSDN does state that if there are build errors, the menu option is not available. I double-check that, and build successfully it did.

Stumped!

Tuesday, 27 April 2010

Visual Studio 2010: Enabling Older DotNet Frameworks (or how I learnt to love multi-targeting)

I've been told that SQL 2005 doesn't support CLR functions created with DotNet Framework 4.0 - I haven't tested it cos When attempting to create a new SQL 2005 CLR project using Visual Studio 2010, I found myself at a loss to work out how to create a .NET Framework 3.0 project ie using the new and improved multi-targeting feature.

The only option available to me in the 'New Project' dialog box was a 4.0 Framework. Plus a handy 'More Frameworks" option which took me to the download page for various dotnet frameworks. I already had 3.0, 2.0 and 1.1 installed on the machine so the need to redownload and reinstall them seemed irritatingly redundant.



So after much googling, I found the solution.

You have to install .NET Framework 3.5.

Ummm... that's it.

Close VS, install 3.5 framework, open VS and wacko!

Contrary to the installer instructions, I did not have to reboot.

Friday, 15 January 2010

SSAS: Vague Processing Errors About Connection Strings Are Not My Friend

Oh Visual Studio, you crazy cat, you! I've just spent the past hour or so going stark raving mad. courtesy of a DBA who's less than 'expert' and a teeny little Visual Studio property I neglected to set properly.


When deploying my Analysis Services database, the processing kept failing with a vague error along the lines of "connection string is invalid" and "login timeout". Now, I know there's nothing wrong with the connection string cos it works fine in VS, right? Wrong!


When deploying a cube to the Integration environment, my DBA told me that I couldn't just deploy it (outrageous suggestion), no I had to give them the project, they would deploy it, then I had to reimport it from the new environment into Visual Studio. WTF? Ok, I'll play. I'm new here after all so who am I to question their processes?


So when I later needed to redeploy some changes I'd since made, the DBA told me I had to do the same again. At this juncture, I put up a big red stop sign and start asking some questions. Turns out they do that so the Impersonation works correctly. Why wasn't it working as expected is the real question they neglected to ask and solve. Instead of fixing the problem, the treated a short-term workaround as the solution, and stopped there, purely because they didn't know any better.


Having rarely used impersonation before I didn't have an immediate answer. I've always just use the service account as my credentials because I'm lazy and frankly I can't really see why, on a DEV box mind you, you'd bother? Whatever, I don't care so much, if that's how they want to do it I'm easy.


So the problem is that after redeploying the project, it wouldn't reprocess. As I mentioned earlier, I was getting these vague errors about logins and connections. Turns out the solution was really freakin' simple.




See that little "Remove Passwords" property? Yeah that's the sucker. That cheeky tyke was set to TRUE. Set it to false and whaddya know? Everything works a treat.

Aaaah... Friday 4pm... and time for a brewskie.

Monday, 12 October 2009

TSQL: Programmatically Create DDL scripts for Foreign Keys

I'll admit it, I'm lazy... I hate typing exercises. I don't need the practice and I'm sure the person paying my somewhat exhorbitant fee doesn't want to fork out for something so mundane either. That means I'm always on the hunt for a shortcut or two, I mean, process improvement.

These two snippets will generate the DDL to create all the foreign keys in the database.
The first will check if it exists first, the second drops it first. Use whichever suits.

SELECT 'IF NOT EXISTS (SELECT * FROM sys.FOREIGN_KEYS WHERE name = ''' + fk.Name + ''' AND Parent_Object_Id = object_id(''['+ s.name + '].[' + t.name + ']'')) BEGIN '
+ char(13) + char(9) + 'ALTER TABLE ' + s.name + '.' + t.name + ' WITH CHECK '
+ char(13) + char(9) + 'ADD CONSTRAINT ['+fk.Name +'] '
+ char(13) + char(9) + 'FOREIGN KEY(['+c.name+']) REFERENCES [' + fks.name + '].['+fkt.name+'] (['+fkc.name+']) '
+ char(13) + 'END'
from sys.foreign_keys fk
inner join sys.foreign_key_columns fkcm on fkcm.constraint_object_id = fk.object_id
inner join sys.tables t on t.object_id = fkcm.Parent_Object_Id
inner join sys.tables fkt on fkt.object_id = fkcm.Referenced_Object_Id
inner join sys.columns c on c.object_id = t.object_id and c.column_id = fkcm.parent_column_id
inner join sys.columns fkc on fkc.object_id = fkt.object_id and fkc.column_id = fkcm.referenced_column_id
inner join sys.schemas s on s.schema_id = t.schema_id
inner join sys.schemas fks on fks.schema_id = fkt.schema_id
ORDER BY 1


And the second one (ie does a drop first)...

SELECT DISTINCT 'IF EXISTS (SELECT * FROM sys.FOREIGN_KEYS WHERE name = ''' + fk.Name + ''' AND Parent_Object_Id = object_id(''['+ s.name + '].[' + t.name + ']'')) BEGIN '
+ char(13) + char(9) + 'ALTER TABLE ' + s.name + '.' + t.Name + ' DROP CONSTRAINT ' + fk.Name
+ char(13) + 'END'
+ char(13) + 'ALTER TABLE ' + s.name + '.' + t.name
+ char(13) + char(9) + 'WITH CHECK ADD CONSTRAINT ['+fk.Name +'] '
+ char(13) + char(9) + 'FOREIGN KEY(['+c.name+']) REFERENCES [' + fks.name + '].['+fkt.name+'] (['+fkc.name+']) '
+ char(13)
from sys.foreign_keys fk
inner join sys.foreign_key_columns fkcm on fkcm.constraint_object_id = fk.object_id
inner join sys.tables t on t.object_id = fkcm.Parent_Object_Id
inner join sys.tables fkt on fkt.object_id = fkcm.Referenced_Object_Id
inner join sys.columns c on c.object_id = t.object_id and c.column_id = fkcm.parent_column_id
inner join sys.columns fkc on fkc.object_id = fkt.object_id and fkc.column_id = fkcm.referenced_column_id
inner join sys.schemas s on s.schema_id = t.schema_id
inner join sys.schemas fks on fks.schema_id = fkt.schema_id
ORDER BY 1

Integration Services Gotcha #6631984: The Table Variable Data Flow Affair

I encountered a teeny little SSIS glitch today that completely did my head in for a good 30 minutes. Ok, I'll admit it, more like an hour.

In my data flow, I had an OLEDB Source which declared and populated a table variable then returned some data, which included a join on my table variable.

The source query worked perfectly, returning plenty of data when tested through SSMS. As soon as I ran the data flow, it would be successful but zero data.

That freakin' table variable was messin' with me. Took it out, replaced it with a subquery and whammo! Data.

Oh SSIS, you pernickety old man, you!

Monday, 20 July 2009

SQL 2008: Using MERGE to do a Type 2 Change

I'm going to do a brief overview of the SQL 2008 MERGE statement. Primarily for my own purposes - being to document the usage for type 2 changes so I don't forget it.

First let's begin with the basic type 1 usage.


All fairly self-explanatory. We have a source and a target. We join the two tables, identify new records, changed records and deleted records. And deal with them appropriately.

So let's move onto the type 2 changes. Here we need to:

  • Insert New Records
  • Disable changed records
  • Add a new record for the changed records
That's 2 inserts and an update. The MERGE won't allow that so we have to nest our merge inside an insert. Luckily this works because MERGE allows output!


Ignore the outer INSERT for the moment. We have a MERGE statement which is very similar to our type 1 MERGE. Compare the Source and Target, INSERT new records as required. The difference being that when the matched record contains a change, instead of updating the changed field, we set the CurrentFlag and the EndDate so as to disable the record.


Of course that means we still don't have a record which contains our updated values. So, now lets turn our attention to the OUTPUT clause.

This will be familiar to anyone who used it in SQL 2005 in much the same context - to identify rows which have been modified/added/deleted. Really the origin of MERGE seems to be strongly rooted in the SQL 2005 OUTPUT clause. On the upside, unlike the SQL 2005 version and UPDATE is an UPDATE and not a DELETE and an INSERT. Small but handy! Anyway, where were we? Let's return the columns we need to create a new record and a special $Action column - this tells us what actually happened ie UPDATE, INSERT or DELETE. We need this because the OUTPUT will return everything that occured in the MERGE.


Now we have a set of rows that have been actioned by the MERGE. Still, we haven't actually done anything with them.

To do that, we enclose the entire MERGE statement, including it's OUTPUT clause of course, into brackets so we can make it the FROM clause for an INSERT. Our OUTPUT clause is return the columns we need for the INSERT remember? So now we just add our INSERT INTO at the front, add an alias to the closed bracket of the FROM clause AND, drumroll... whack a WHERE clause on the end. This where clause just filters the OUTPUT from the MERGE so we only insert new records for those that were updated, ignoring those that may have been inserted etc in the MERGE.

Easy!


-------------------------------------------------------------------------------------------------

For the purposes of copy/paste-ability, here is a text version of the TSQL. Images have been used above for improved readability.

--A simple type 1 change
MERGE INTO DimCustomer c
USING StageCustomer stg ON c.CustomerId = stg.CustomerId
WHEN NOT MATCHED BY TARGET THEN --ie new
INSERT (CustomerId,CustomerName,IsActive) VALUES (
stg.CustomerId
,stg.CustomerName
,stg.IsActive
)
WHEN MATCHED AND c.CustomerName <> stg.CustomerName THEN --ie changed
UPDATE SET c.CustomerName = stg.CustomerName
WHEN NOT MATCHED BY SOURCE THEN --ie deleted
DELETE


--a type 2 change
INSERT INTO dbo.DimCustomer
SELECT CustomerId, CustomerName, IsActive, IsCurrent, Eff_Date, End_Date
FROM
( MERGE DimCustomer c
USING StageCustomer stg ON c.CustomerId = stg.CustomerId
WHEN NOT MATCHED THEN INSERT VALUES (
stg.CustomerId
,stg.CustomerName
,stg.IsActive
,1 --> IsCurrent
,GETDATE() --> Eff_Date
,NULL --> End_Date
)
WHEN MATCHED AND c.IsCurrent = 1 AND (c.CustomerName <> stg.CustomerName) THEN UPDATE SET
c.IsCurrent = 0
,c.End_date = GETDATE()
OUTPUT $Action as RowAction
,stg.CustomerId
,stg.CustomerName
,1 AS IsCurrent
,GETDATE() as Eff_Date
,NULL as End_Date
) m
WHERE m.RowAction = 'UPDATE'

Tuesday, 9 June 2009

SQL: Retrieving the Metadata for a Query

Sometimes you need to know what the metadata for a query looks like before runtime. This is particularly relevant with SSIS.


I use it rarely but every time I do have a need for it, I can never remember the name of the option and it takes me an hour at least to track it down again. So I'm hereby cementing it in stone so I don't forget it again!

SET FMTONLY ON

That's the one. It will return the columns and data types to the query engine. Note: it does not actually run the query or return any results. It's purely to return metadata about the query.

It's surprisingly handy from time to time.

Wednesday, 3 June 2009

SQL 2005: Scalar Resultsets and SSIS Variables

Found an interesting SSIS tidbit today which I've amazingly never encountered before.  Not quite sure how I've never managed to hit this problem before - it seems surprising to me but there you go, I guess it happens from time to time.

In an ExecuteSQLTask, I was returning a scalar value to a variable.  The value returned was a bigint.  My SSIS variable was a bigint.   The step failed with an invalid type error.  What the?  Changed the type to an object.  Success this time but not much help to me down the control flow because I can't read it without a ForEach - which I really don't wanna do for a scalar value.  I mean that would just be silly.

According to Technet "With an OLE DB or Excel connection manager, the connection manager explicitly converts values of the following types, DBTYPE_NUMERIC, DBTYPE_GUID, and DBTYPE_BYTES, to strings".  Seems crazy to me!  I changed the SSIS variable to a string and what do you know, success.

Nuts!

Friday, 24 April 2009

SQL 2008 SP1 - a must have!

Some time ago I installed SQL 2008 on a VPC which was also running MOSS and PerformancePoint.  Soon enough the VPC started to run out of memory.  I tore my hair out for a few days because nothing I did made a good dent in the problem.  


Till I found the silver bullet.

Consistently SSAS was high on the list of memory-hungry services - plateauing around the 500Mb mark.  Yeah, that's a lot.  With only a couple of cubes and no queries running it did seem to be hugely excessive. 

What to do?  Install SP1.

That's it.

Usage dropped down to somewhere around 30-80Mb.  Much more memory friendly!

Interestingly, of the many bug fixes included, in SP1 (full list),  a memory leak in SSAS is not one of them! 

Thursday, 9 April 2009

TSQL: Truncating Datetime

It crops up all the time.  You have a datetime column.  In your select, you don't care about the time, you only care about the date because you want to group all records for a particular date, for example.  So what's the quickest way to go about it?


Well let's start with the options...
  1. CONVERT
  2. FLOOR
  3. DATEADD/DATEDIFF
If you are using SQL 2008, you could add another option and CAST to DATE.  Perfect if the recipient of the data will accept it.  I'm gonna assume however, as is most likely at this point in time, that you aren't using SQL 2008 yet as that does seem to still be the case in most organisations.  I'm also going to assume that you need to return a datetime.

So, to option 1: CONVERT.

A popular choice because it's easily readable.  Let me give you an example.

CAST((CONVERT(varchar(11),getdate(),120)) as DATETIME)
This will return 2009-04-09 00:00:00.000

Option 2:  FLOOR

CAST(FLOOR(CAST(Getdate() as FLOAT)) as DATETIME)
Returns 2009-04-09 00:00:00.000

Obviously faster than convert because we aren't working with any strings.

Option 3: DATEADD

DATEADD(dd, DATEDIFF(dd, 0 getdate()), 0)
Returns 2009-04-09 00:00:00.000 (shock horror!)

Guess what?  This version is even faster again because we don't have any conversion to do at all.

To what degree are these options faster?

Well, a quick test by plonking a million records into a temp table which contains only a datetime column showed my execution times to be fairly similar but my CPU times  were increasingly quicker.

CONVERT - 1591
FLOOR - 434
DATEADD - 382

The convert is horrible obviously and should never be considered even at the expense of readability.  Clearly there's not a vast difference between FLOOR and DATEADD but I guess I like the DATEADD method for it's lack of conversion as well as it's performance gain, however small.  It just seems cleaner to me.

Friday, 6 February 2009

SQL 2005 SSIS: Unpivot Transformation - Destination Column

Came across an interesting little bug today in SSIS...

When using the Unpivot transformation, if you attempt to see the results of the Destination Column in a data viewer well, you can't... the column just isn't there. However, when the output is sent to an actual data destination it is created as expected. You just have no way of knowing that until you actually look at the destination itself.

Wasted quite a bit of time on that one!

Monday, 22 December 2008

SQL 2005 SP3 has landed

SP3 for SQL 2005 is now available for download.

Included are a number of Bug Fixes , performance enhancements for using SSRS in SharePoint Integration mode, improved rendering of PDF fonts in SSRS, support for SQL 2008 Dbs with SQL 2005 Notification Services and support for Teradata Dbs as a data source for Report Models.

This Service Pack is cumulative and can be run on any version of SQL 2005.

Tuesday, 4 November 2008

Microsoft Azure.... the blue loo

Microsoft announced it's Azure service last week. In their words Azure is a tool that allows you to "Build new applications in the cloud - or use interoperable services that run on Microsoft infrastructure to extend and enhance your existing applications". From our, the BI developers', perspective Azure is hosting not just data but also Visual Studio solutions ie developing and deployment within the cloud.

Not that it's limited to Microsoft tools and environments of course.

So I immediately joined the CTP for SQL Services. I've downloaded the SQL Data Services SDK and have been placed on a waitlist for access to the cloud.

From what I can tell so far, you can use REST or SOAP for CRUD operations and LINQ for queries. Interestingly, there is no 'schema' as such - new properties can be added to entities as required - no need to add a new 'column'. What that means in practical terms I'm yet to find out. I suppose it will become apparent once I can access the cloud.

More info will follow shortly.

Tuesday, 28 October 2008

What’s New in SQL 2008 – Part 3 – Table Type

The new Table type allows you to save a table definition in the database which can be reused later to define table variables, stored procedure parameters and functions. To create a Type, we use the CREATE TYPE clause, for example:

CREATE TYPE dbo.ProductList as TABLE (

    ProductName varchar(200) NULL,

    ProductCategory varchar(200) NOT NULL)


Important Notes:

  • The metadata is visible using sys.table_types
  • Variables/parameters cannot be set to NULL – if no table is supplied, an empty table is generated
  • Not affected by transaction rollbacks
  • Do not cause recompilations


Examples

Create Table-Type Variables

DECLARE @t as dbo.ProductList;
INSERT INTO @t VALUES
('Grey Slate Paver 10x10','Paver'),
('Blue Slate Paver 10x10','Paver'),
('Heritage Blue 12x6','Brick')

Creating a Stored Procedure with a Table-typed Parameter

CREATE PROC dbo.uspGetOrdersforProduct (
@t as dbo.ProductList READONLY,
@StartDate as Date,
@EndDate as Date
) AS
SELECT o.OrderId,
o.OrderDate,
o.CustomerId,
o.ProductId,
o.Amount,
o.Qty
FROM tOrder as o
INNER JOIN tProduct t on t.ProductId = o.ProductId
INNER JOIN @t as f on f.ProductCategory = t.ProductCategory and (t.ProductName is NULL or t.ProductName = f.ProductName)
WHERE o.OrderDate between @StartDate and @EndDate

Passing Table-parameters into Stored Procedures

DECLARE @t as dbo.ProductList;
INSERT INTO @t VALUES
(NULL,'Paver'),
(NULL,'Brick')

EXEC dbo.uspGetOrdersforProduct @t

What’s New in SQL 2008 – Part 2 – Data Types

Date and Time

Data Type

Size (bytes)

Example

Notes

Client Providers

DATE

3

'2009-10-01'

OLEDB: DBTYPE_DBDATE
ADO.Net: DateTime
ODBC: SQL_DATE

TIME

3-5

'13:12:00.1234567'

Accurate to 100 nanoseconds

OLEDB: DBTYPE_DBTIME
ADO.Net: TimeSpan
ODBC: SQL_TIME

DATETIME2

6-8

'2009-10-01 13:12:00.1234567'

Accurate to 100 nanoseconds
Can specify precision ie decimal places eg DATETIME(3)

OLEDB: DBTYPE_DBTIMESTAMP
ADO.Net: DateTime
ODBC: SQL_TIMESTAMP

DATETIMEOFFSET

8-10

'2009-10-01 13:12:00.1234567+02:00'

Accurate to 100 nanoseconds

OLEDB: DBTYPE_DBTIMESTAMP
ADO.Net: DateTime
ODBC: SQL_TIMESTAMP


There are a number of new functions which can also be used with these new datatypes including:

  • SYSDATETIME (returns DATETIME2), SYSUTCDATETIME, SYSDATETIMEOFFSET (includes time zone in return value) – Current Date and Time
  • SWITCHOFFSET – to change the timezone for a DATETIMEOFFSET value
  • TODATETIMEOFFSET – convert non-timezone-aware values to time-zone-aware

Other

Data Type

Size (bytes)

Useage

Returns

Notes

HIERARCHYID

VARBINARY
eg 5 bytes for 100k records at 6-Levels

DECLARE @h hierarchyid = hierarchyid::GetRoot()

DECLARE @c hierarchyid

SET @c = @h.GetDescendant(NULL, NULL)
SELECT @c.ToString()

'/1/'

Represents position in a hierarchy

Actually a CLR function with several methods eg

  • GetDescendant/IsDescendantOf
  • GetLevel/GetRoot
  • GetReParentedValue
  • ToString/Read/Write (CLR)

What's New in SQL 2008 - Part 1 - Functionality


Function

Usage Example

Notes

Declare & Initialise

DECLARE @iValue as integer = 100;


Compound Assignments

SET @iValue -= 50

UPDATE dimCustomer SET Balance += 200

+=

-=

*=

/=

%=

Multi-Row Inserts Using VALUES

INSERT INTO dimProduct (Label, BusinessKey,LastUpdatedBy,LastUpdatedDate) VALUES
('Grey Slate Paver 10x10',1111,'dbo','20091001 13:12:00'),
('Blue Slate Paver 10x10',1112,'dbo','20091001 13:12:00'),
('Black Slate Paver 10x10',1113,'dbo','20091001 13:12:00')


The insert is executed as a single transaction.

Interestingly, it can also be used like this:

SELECT * FROM (VALUES
('Grey Slate Paver 10x10',1111,'dbo','20091001 13:12:00')
('Blue Slate Paver 10x10',1112,'dbo','20091001 13:12:00')
('Black Slate Paver 10x10',1113,'dbo','20091001 13:12:00')
) as p (Label, BusinessKey,LastUpdatedBy,LastUpdatedDate)

MERGE

MERGE INTO dbo.dimProduct as d
USING dbo.dimProductStage as s ON d.ProductKey = s.ProductId
WHEN MATCHED THEN
UPDATE SET
d.ProductName = s.ProductName,
d.ProductCategory = s.ProductCategory
WHEN NOT MATCHED THEN

INSERT (ProductName,ProductCategory)

VALUES (s.ProductName, s.ProductCategory)

WHEN NOT MATCHED BY SOURCE THEN DELETE
OUTPUT $action,
deleted.ProductId as DeletedId,
inserted.ProductId as InsertedId;

Grouping Sets

SELECT Year(OrderDate) as OrderYear,
SalespersonId,
OrderType,
count(OrderId) as OrderCount,
sum(Amount) as OrderAmount
FROM factOrders
GROUP BY GROUPING SETS (
(OrderType, Year(OrderDate)),
(SalespersonId,OrderType, Year(OrderDate)),
(SalespersonId, Year(OrderDate)),
(Year(OrderDate)),

());

() empty brackets represents an 'ALL' grouping


Filtered Indexes

CREATE NONCLUSTERED INDEX idxOrders_Product
ON factOrders(OrderId,ProductId)
WHERE CreditFlag IS NULL

Change Data Capture


** to be discussed in further detail in an upcoming post **

Tracks data changes into a table.
Enabled using sys.sp_cdc_enable_db or sys.sp_cdc_enable_table
Disabled using sys.sp_cdc_disable_db or sys.sp_cdc_disable_table
Uses SQL Agent


Monday, 25 August 2008

Interesting Read

Great article comparing the PIVOT (SQL 2005+) function with doing it the old-fashioned way! Take a read...

http://www.sqlservercentral.com/articles/T-SQL/63681/

Thursday, 3 July 2008

SQL: Concatenating Rows into a Scalar Value

This is an issue which pops up from time to time, often when passing values back to .Net applications.

The Problem
Taking values in rows and creating a single string value containing a delimited list of all those values.

The Old Solution
The method I have used in the past is to use the FOR XML clause and build a string that way. It works.

DECLARE @DepartmentName VARCHAR(1000)
SELECT @DepartmentName = Name + ','
FROM AdventureWorks.HumanResources.Department
WHERE GroupName = 'Executive General and Administration'
FOR XML PATH ('')
SELECT @DepartmentName AS DepartmentNames


The Sexier Solution
This idea comes courtesy of Ken Simmons at SQL Tips... so simple and I'm kind of annoyed that I've never tried it myself!

DECLARE @DepartmentName VARCHAR(1000)
SELECT @DepartmentName = COALESCE(@DepartmentName,'') + Name + ';' 
FROM AdventureWorks.HumanResources.Department
WHERE GroupName = 'Executive General and Administration'
SELECT @DepartmentName AS DepartmentNames


What Were the Actual Results from Testing?
Sadly, the sexier solution under-performs in comparison to the FOR XML solution.

MethodMillisecondsBytesReceivedRows Returned
FOR XML7898913
COALESCE2651050560921


Obviously this lag is due to the number of rows which are returned to the client. Based on the stats, the FOR XML does all it's aggregation and concatenation server-side whereas the COALESCE method does it client-side.

So I guess that means the old way is still the best. Or is it? I wonder in what other circumstances COALESCE could be used? Ok so it under-performed in this instance but I suspect there are other uses which could be of great value. Got any ideas?