Happy New Year 2012!

As 2011 is winding down, it’s time to reflect on the past and plan for the future. 2011 has been a very exciting year for Microsoft BI and me.

  1. Gartner positioned Microsoft as a leader in the 2011 Magic Quadrant for Business Intelligence Platforms.
  2. Although SQL Server 2012 will technically ship early next year, we can say it’s a done deal as it’s currently in a release candidate phase. The most important news from a BI perspective is the evolution of the Business Intelligence Semantic Model (BISM), which an umbrella name for both Multidimensional and Tabular models.
  3. The Tabular model provides us with a nice personal (PowerPivot for Excel)-team (PowerPivot for SharePoint)-organizational (Analysis Services Tabular) continuum on a single platform.
  4. Power View extends the BI reporting toolset with a sleek web-based reporting tool for authoring highly interactive and presentation-ready reports.
  5. In its second release, Master Data Services (MDS) comes out of age and now allows end users to use Excel to manage master data. The newcomer, Data Quality Services (DQS), complements MDS nicely in the never-ending pursuit for clean and trusted data. Integration Services has also nice enhancements. Finally, columnstore indexes will help to aggregate large datasets, such as the scenario I mentioned in this blog.

Looking forward to 2012 and beyond, here is my top 5 BI wish list:

  1. Extending the Tabular capabilities with more professional features, such as scope assignments, role-playing dimensions, MDX query support, and so on, to enhance its reach further in the corporate space. Ideally, I expect at some point in future unification of Multidimensional and Tabular so BI pros don’t have to choose a model.
  2. Extending Power View to support multidimensional cubes. Further, in the reporting area, I expect an embeddable web-based OLAP browser (it’s time for Dundas Chart to come back to live) and an improved MDX query designer (no, I haven’t lost hope for this one).
  3. Enhanced Excel BI capabilities so Excel becomes the BI tool of choice. This includes supporting PowerPivot natively and overhauling the reporting capabilities beyond the venerable PivotTable and PivotChart. Ideally, what I am hoping for is decoupling Power View from SharePoint and integrating it with Excel and custom web applications. Power View is too cool to be confined onlyin SharePoint.
  4. Extending Microsoft Azure with BI capabilities to let solution providers host BI models in the cloud.
  5. Bringing BI to mobile devices.

On the personal side of things, I’ve been fortunate to stay healthy and busy (very busy). The Atlanta BI group, which I am leading, has grown in size and we now enjoy having 40-50 people attending our monthly meetings. For the past few months, I’ve been working on my next book, Applied Microsoft SQL Server 2012 Analysis Services (Tabular Modeling), which I expect to get published in March. And, my consulting business has been great!

I wish you a healthy and prosperous year! I hope to meet many of you in 2012. Meanwhile, you can find me online at the usual places: www.prologika.com | blog | linkedin | twitter.

Happy New Year!

Using the Hash Group Hint to Speed up ColumnStore Indexes

As I mentioned in my blog post on this subject, I’ve found a good use of SQL Server 2012 columnstore indexes to speed up significantly ETL processes that need to aggregate large datasets. But we run into a snag, which got promoted to a bug by Microsoft Support Services. Under some conditions, SQL Server would create a plan that uses a “stream aggregate” operator instead of the more efficient hash match aggregate. This is illustrated by the following plans.

This plan uses the stream aggregate and the query is much slower. If you hover on the Sort predicate, you will see a warning that the sort will spill data to tempdb.

6266.p1.png-550x0

By contrast, this plan uses the Hash Match predicate and the query is about three times faster.

1464.p2.png-550x0

As I mentioned, this appears to be a bug with SQL Server 2012, which might not get fixed in RTM. Meanwhile, force your queries to use the HASH GROUP query hint to force SQL Server to use a hash match with columnstore indexes.

GROUP BY              Organization_Id,

                      Item_Profile_Id,

            …

OPTION (HASH GROUP)

Upgrade Issues with PerformancePoint 2010 Filters

Issue: Migrate a PerformancePoint dashboard from MOSS 2007 to SharePoint 2010. We used the Import PerformancePoint Content feature to import existing dashboards from the PerformancePoint database and this saved us a lot of effort. However, filters failed with this rather obscure error:

121611_0116_UpgradeIssu1

After tracing with SQL Profiler and looking at the Windows Event log, we saw the following almost as useless error):

Microsoft.AnalysisServices.AdomdClient.AdomdErrorResponseException: Query (1, 7) Parser: The syntax for ‘{‘ is incorrect.

In this case, the dashboard is using a Time Intelligence filter mapped to an Analysis Services dimension.

Resolution: The upgrade process has mapped the filter’s formula in the connection to the scorecard. We fixed the issue by mapping the filter data source (not formula), as shown in the screenshot below. In this case, the CompareDay is the data source name.

121611_0116_UpgradeIssu2

Columnstore Indexes To Speed ETL

SQL Server 2012 introduces columnstore indexes. Using the same in-memory VertiPaq engine that powers PowerPivot and Analysis Services Tabular, columnstore indexes can speed up dramatically SQL queries that aggregate large datasets. For a great introduction to columnstore indexes, see the video presentation, “Columnstore Indexes Unveiled” by Eric Hanson. I personally don’t see columnstore indexes as a replacement of Analysis Services because an analytical layer has much more to offer than just better performance. However, in a recent project we’ve found a great use of columnstore indexes to speed up ETL processes.

Issue: Perform an initial load of a snapshot fact table for inventory analysis from another fact table with one billion rows. For each day, extract some 200 million rows from the source fact table and group these rows into a resulting set of about 300,000 rows to load the snapshot fact table for that day. The initial estimates indicated that that the extraction query alone takes about 15 minutes when using a clustered index. And, that’s just for one day. Given this speed, we estimated the initial load could take weeks.

Solution: We upgraded to SQL Server 2012 and created a columnstore index on selected columns from the source fact table. We excluded high-cardinality columns that were not used by the extraction query to reduce the size of the index. Creating the index on the source table (1 billion rows) took about 10 minutes and this is very impressive. The disk footprint of the index was about 4GB. We ran the same extraction query and saw a five-fold performance improvement. This query, which would previously run for 15 minutes with a B-tree clustered index, would now finish in 3 minutes with a columnstore index.

When testing your queries with columnstore indexes, it is important to make sure that the query executes in Batch mode. As you can see in the screenshot below, the query uses the columnstore index and the execution mode is Batch. If it says Row, the query performance degrades significantly. Watch Eric Hanson’s presentation to understand why this happens and possible workarounds.

120811_0048_Columnstore1

When I did initial tests to test the columnstore index, I ran into a gotcha. I tested a simple “SELECT SUM(X) FROM TABLE” query to find that it executes in a Row mode and the query took about 2 minutes to finish. As usual, the first thing I try doesn’t work. As it turned out, currently a columnstore index doesn’t support batch mode with scalar aggregates. You need to rewrite the query with a GROUP BY as Eric Hanson explains this in more details in his blog, “Perform Scalar Aggregates and Still get the Benefit of Batch Processing”. This is rather unfortunate because every ad-hoc report starts with the end user dropping a measure and the report tool generating such queries.

When testing the performance gain, it’s useful to compare how the same query would perform without a columnstore index. Instead of having two tables or dropping the index, you could simply tell SQL Server to ignore the columnstore index, such as:

SELECT … FROM…WHERE…GROUP BY…

OPTION (IGNORE_NONCLUSTERED_COLUMNSTORE_INDEX)

 

To sum up, when you are pushing the envelope of traditional B-tree indexes and queries aggregate data, consider SQL Server 2012 columnstore indexes. The query optimizer would automatically favor a columnstore index when it makes sense to use it. Columnstore indexes require a read-only table but the cost of dropping and recreating them is not that high. Or, you could add them to speed up the initial DW load and drop them once the load completes.

 

What’s New in Power View RC0

I was just about to write a blog about the new features in the Release Candidate (RC) build of Power View and I saw that Robert had written a great blog already. It’s great to see the product coming along so nicely. Can’t wait to be able to use multidimensional cubes as data sources!

One feature that stirred a lot of excitement and deserves more attention is Power View export to PowerPoint. When you export a report to PowerPoint, initially you get static slides that show images of the report pages.

112311_0129_WhatsNewinP1

Don’t be fool however as you can click the Click to Interact button to render the report live. The interactive mode preserves all report interactive features. For example, in the screenshot below I am playing the scatter chart animation.

112311_0129_WhatsNewinP2

Behind the scenes, the interactive mode uses a Silverlight alternative hosting control (right-click the object on the slice and click View Code) that points to the deployed report on the SharePoint server. Therefore, a live connection and permissions to SharePoint are required to preview the report inside PowerPoint.

112311_0129_WhatsNewinP3

UPDATE 11/29/11

As explained in the SQL Server 2012 RC0 release notes, the pre-release bits of Silverlight 5 are not available in 64-bit. The RTM release will include a 64-bit installer and it looks like it be available pretty soon. Meanwhile, trying to interact with a Power View report exported to PowerPoint 64-bit wil fail with the error “Some controls on this presentation can’t be activated. They might not be registered on this computer.”

 

SQL Server 2012 RC is Here

The public release of Microsoft SQL Server
2012 RC0 is now available.

PowerPivot for Excel SQL Server 2012 RC0 is Here

Microsoft just published the RC0 (release candidate) bits of PowerPivot for Excel SQL Server 2012. Kudos for Microsoft for making the RC bits public this time. Stay tuned for an announcement for the arrival of the SQL Server 2012 RC bits soon. The RC build is a feature-complete and stable build. This will be the last pre-release build until SQL Server 2012 officially ships next year.

Here are all SQL Server 2012 RC0 links pulished so far:

Microsoft SQL Server 2012 Express RC0

Microsoft SQL Server 2012 Semantic Language Statistics RC0

Microsoft SQL Server 2012 Master Data Service Add-in for Microsoft Excel 2010

Microsoft SQL server 2012 Manageability Tool Kit RC0

Report Server Settings in SQL Server 2012 SharePoint Integration Mode

111611_0119_ReportServe1Well, rules have changed. As Jaime Tarquino from the SSRS team explained in his blog, “SQL Server 2012 Reporting Services SharePoint Integrated Mode”, there is no rsreportserver.config file anymore moving to Reporting Services 2012 in SharePoint integration mode (there are no configuration changes with native mode). Instead, the configurations sections are now saved in the SharePoint configuration database.

Problem: A customer plans to upgrade to SQL Server 2012 and SharePoint integration mode. They had the following custom renderer registered in the rsreportserver.config file that uses device information settings to customize the Excel renderer to suppress report headers and use Excel headers instead:

<Extension Name=EXCELNOHEADER Type=Microsoft.ReportingServices.Rendering.ExcelRenderer.ExcelRenderer,Microsoft.ReportingServices.ExcelRendering>

    <OverrideNames>

        <Name Language=en-US>Excel (no Header)</Name>

    </OverrideNames>

    <Configuration>

        <DeviceInfo>

            <SimplePageHeaders>true</SimplePageHeaders>

            <RemoveSpace>0.012in</RemoveSpace>

        </DeviceInfo>

    </Configuration>

</Extension>

The customer claimed that this configuration is so important that there is no way they could upgrade if this doesn’t work in SQL Server 2012. And after ensuring them that this is an easy fix, I’ve found myself spending hours to understand what needs to be done. On the positive side, I’ve picked up a few PowerShell skills along the way which may come handy given that PowerShell is omni-present nowadays.

Solution: In SQL Server 2012 (SharePoint integrated mode), you need to make such changes with PowerShell. The PowerShell Cmdlets topic in BOL is a good writeup about the SSRS cmdlets. The following script gets the job done:

Add-PSSnapin Microsoft.SharePoint.PowerShell

$apps = Get-SPRSServiceApplication

foreach ($app in $apps)

{

New-SPRSExtension -identity $app -ExtensionType “Render” -name “EXCELNOHEADER” -TypeName “Microsoft.ReportingServices.Rendering.ExcelOpenXmlRenderer.ExcelOpenXmlRenderer,Microsoft.ReportingServices.ExcelRendering” -ServerDirectives “<OverrideNames><Name Language=’en-US’>Excel (no Header)</Name></OverrideNames>” -ExtensionConfiguration “<DeviceInfo><SimplePageHeaders>true</SimplePageHeaders><RemoveSpace>0.012in</RemoveSpace></DeviceInfo>”

}

The script starts by importing the Microsoft.SharePoint.PowerShell snapi (you don’t need it if you run the remainder of the script in the SharePoint 2010 Management Shell). Then, the Get-SPRSServiceApplication cmdlet returns all the Reporting Services Service Applications available in the farm. For each application, I call the New-SPRSExtension to register the new renderer. To run the script:

  1. Remote to the SharePoint server. You must have SharePoint Farm Administrator rights to run the script.
  2. Open SharePoint 2010 Management Shell from the Microsoft SharePoint 2010 Products program group. Or, if you have installed the Windows PowerShell ISE feature, you can open Windows PowerShell ISE from the Accessories program group but make it sure to run it as administrator (right-click on it and click Run As Administrator).
  3. Copy and paste the script, and hit Enter to run it. The figure below shows the script in Windows PowerShell ISE:
  4. Do iisreset to apply the configuration changes.

For a full list of the New-SPRSExtension cmdlet parameters, you can execute the following command

Get-help New-SPRSExtension – full 

To verify that the changes are applied you could use one of the following two approaches:

  1. Execute the following script to output the configuration of all extensions:
    $apps = Get-SPRSServiceApplication

foreach ($app in $apps) {

    $extensions = Get-SPRSExtension -identity $app

    foreach ($extension in $extensions) {

echo $extension.ExtensionType $extension.Name

echo $extension | select -ExpandProperty ConfigurationXml }

2. In SSMS, connect to the SharePoint config database and run the following query. Examine the content of the Properties column to make sure that the custom extension is registered.

SELECT * FROM [dbo].[Objects] WHERE properties like ‘%excelnoheader%’     

I was tempted to try modifying directly the Properties column in the Objects table and add the new extension section but I realized I had better use of my time than troubleshooting failed SharePoint installations so I decided to stick with the supported way.

Special thanks to Prash Shirolkar from the Reporting Services team for shedding light in dark places. His blog has valuable insights about Reporting Services and SharePoint.

MDXParameter Sample

MDXParameter is a nice little utility that captures MDX queries sent from SSRS reports and replaces parameter placeholders with the actual values. I’ve been using the excellent Darren Gosbell’s Query Capture sample which is one of the first utilities I install when starting a new project. MDXParameter has a few more features that you might find appealing, including saving the queries in a database and showing the parameters separately. Installing MDXParameter is simple:

  1. Download MDXParameter from Codeplex and unzip it.
  2. MDX Parameter requires a SQL Server database to capture the queries. It could have been nice to include an option to write the queries to a file or just the screen but currently you must create an empty SQL Server database. You can name the database anything you want.
  3. Double-click MDXParameter.exe to start it. You will be prompted to specify the connection details. Note that MDXParameter will automatically create the table schema.

111111_1320_MDXParamete1

  1. Click OK to launch the main interface and click Start to start the trace.
  2. Go to your SSRS report and run it. Make sure to delete the *rdl.data file in the SSRS project folder to avoid running the report with cached datasets.
  3. Click Stop to stop the trace. You should see the all report MDX queries captured. Select a query to see its statement in the MDX tab. If the query has parameters, MDXParameter will replace them with the actual values (if the Replace Parameter checkbox is selected). This of course is the main reason why you need MDXParameter as it saves you a great deal of time if you want to execute the query with many parameters in SSMS. The tabParametersGrid shows you a grid with the actual parameter values. The Execute MDX button is currently not functioning.

111111_1320_MDXParamete2

MDXParameter supports SSAS 2005, 2008, and R2.

Stopping SSRS Parameter Refresh

[View:https://prologika.com/CS/cfs-file.ashx/__key/CommunityServer.Blogs.Components.WeblogFiles/blog/5736.Product-Sales.rdl:550:0]

Problem: How do we stop the maddening parameter refresh for reports that use SSAS cubes? As you’ve probably found, nothing seems to stop it and the Never Refresh option on the Advanced parameter properties is useless.

Solution: Use the OLE DB provider for Analysis Services for the parameter queries, as shown in the attached report.