Presenting at SQL Saturday BI Atlanta

Come and join me on Saturday, January 9 th at the first SQL Saturday BI edition in Atlanta. You’ll learn about the exiting new BI changes coming to SQL Server 2016 and the Microsoft on-premises roadmap!

The Best Self-Service BI Tools of 2015

I came across this interesting PC Magazine article that just came up to compare 10 popular self-service BI tools. And the winner is? Power BI, of course, rubbing shoulders with Tableau for the Editor Choice award! The author, David Strom, did a great job reviewing the tools (this is not a trivial undertaking) but a few Power BI conclusions deserve clarifications:

  • Cons: “Cloud version has a subset of features found in Windows version” – The cloud version is meant to be simple on purpose so that business users can start analyzing data without any modeling.
  • Sharing: “Microsoft relies on the shared Microsoft OneDrive at Microsoft cloud service (or what it calls a “content pack”) to personalize and share your dashboard and reports via unique URLs” Power BI doesn’t rely on OneDrive for collaboration. Instead it supports three ways to share content: simple dashboard sharing, workspaces, and content packs.
  • Custom visuals: “You can get quickly up to speed by searching through an online visualizations gallery to find the particular presentation template you want to use to show your data. This is the reverse of what many BI tools such as Tableau Desktop ($999.00) at Tableau Software and Domo ($2,000.00) at Domo have you do, and it takes a bit of getting used to.” Not sure what this refers to. There are built-in visualizations and starting up with them is no different than using other tools. But we have also custom visuals that no other vendor has.
  • Custom visuals:” A new section called “Developer Tools” lets you build custom visualizations using a Visual Basic-like scripting language that is documented in a GitHub project. While it is still in beta, it could be a very powerful way to add your own custom look to your dashboards” The Dev Tools for implementing custom visuals outside Visual Studio is in preview but the actual visualization framework is not. And developers use TypeScript (superset of JavaScript) and not Visual Basic.

Speaking about reviews, here are some important Power BI characteristics that make it stand above the rest of the pack:

  1. Data engine and DAX – no other tool can come close to the Power BI in-memory engine that allows data analysts to build data models that are on a par with professional models.
  2. Hybrid architecture that allows to connect your visualizations to on premise data sources.
  3. Self-service ETL with Power Query – as far as I know, no other tool has such capabilities.
  4. Open architecture that allows developers to extend the Power BI capabilities
  5. Great value proposition that follows the freemium model – Power BI Desktop is free, Power BI Mobile is free, Power BI service is mostly free.

pcmag

Getting Rid Of Custom Visuals

Scenario: You might have imported a custom visual in Power BI Desktop, tested it, and decided not to use it. However, even if your reports don’t use the visual anymore, Power BI will still prompt you if you want to enable custom visual with “This report contains a custom visual not provided by Microsoft…”. This is a security warning to avoid malicious code because custom visuals are deployed in Javascript.

Currently, there is no way in Power BI to disable this prompt. To make things worse, neither Power BI Service nor Power BI Desktop have a feature to get rid of the custom visual once it’s added to a Power BI Desktop file.

Solution: Here are the manual steps are followed to get rid of custom visuals in Power BI Desktop file for good:

  1. Copy the Power BI Desktop (*.pbix) file. Rename the file to have a zip extension, e.g. from Adventure Works.pbix to Adventure Works.pbix.zip.
  2. Unzip the file.
  3. In the uncompressed folder where you unzip the file content, navigate to the Report folder, and open the Layout file in your favorite text editor.
  4. At the top of the file content, find a resourcePackage string that includes the visual name (you could search for the name of the visual to locate it). For example, the resourcePackage element might look like this for the Sparkline visual:
    ,”resourcePackages”:[{“resourcePackage”:{“name”:”Sparkline1444636326814″,”items”:[{“path”:”icon.png”,”type”:3},{“path”:”Sparkline.js”,”type”:0}
  5. Carefully, delete this entire string but make sure that you don’t end up with two commas or you miss a comma after the deletion.
  6. While you’re in the uncompressed file content, delete also a folder that has the same name as the visual. Strictly speaking, this step is not needed to avoid the prompt but it’s a good idea to clean up all visual files so that you don’t distribute the visual Javascript source.
  7. Zip the entire content again. For some obscure reason, besides getting rid of the visual, in my case compressing the file reduce the PBI Desktop file size almost twice! This reduces the time to upload the visual to Power BI Service.
  8. Rename the file back to the original file name without the zip extension.

Now when you deploy the PBI Desktop file to Power BI and view its reports, you shouldn’t get prompted anymore.

Implementing User Friendly Names in Tabular

Scenario: You’d want to have user-friendly field names in Tabular, Power Pivot, and Power BI Desktop, such as Claim Amount as opposed to database column names, such as ClaimAmount or Claim_Amount. Multidimensional has a feature that automatically delimits words with spaces when it detects a title case or underscore but Tabular lacks this feature. While you can rename fields in Tabular on field at the time, each step requires a commit action, thus taking long time to rename all fields.

Solution: While I’m not aware of a tool for renaming fields, the following approach should minimize the tedious work on your part:

  1. Wrap your table with a SQL view. It’s a good practice anyway.
  2. Alias the table columns. If you have a lot of columns, the easiest way to alias your columns is to use vertical copy and paste.
    1. In SSMS, script the table as SELECT TO. This generates the SELECT statement in a suitable format for the next steps (column names enclosed in square brackets, comma on the left).
    2. Hold the Alt key and select all columns by doing a vertical selection to enclose all column names, excluding the commas.
    3. Press Ctrl-C to copy.
    4. Hold the Alt key again. Click a place to the right of the first column and drag the mouse cursor down until you reach the last row where the last column is. You should see a vertical line going down.
    5. Type ” AS ” without the quotes. The net effect is that SSMS enters AS for each column.
    6. Press Ctrl-V to paste the column names. Here is the net result:
  3. Now you can delimit the words with spaces. But if you have many columns, this can quickly get tedious too. Enter regular expressions.
  4. Hold the Alt key again for vertical selection and select all alias columns, excluding “AS”.
  5. Press Ctlr-H to bring up the SSMS Find & Replace. In the Find field, enter the regular expression ~(\[)[A-Z]. This expression searches for any capital letter after the left square bracket [.
  6. In the Replace field, enter ” \0″ without the quotes. Notice that these is a space before the backslash. This replaces the capital letter match with an empty space and the capital letter.
  7. Check the “Match Case” and “Use Regular Expressions”. Make sure that the “Look In” drop-down is set to Selection to avoid apply the replace to all the text.
  8. Click Replace All. Now you have all words delimited.

The regular expression I use is not perfect. It won’t discriminate capital letters; for example, it will delimit consecutive capital letters, such as ID as I D, but it’s faster to fix the exceptions than doing all replacements manually. And if you end up with a better expression, please send it my way. The last step, of course, is to import the view and not the table in Tabular, Power Pivot, or PBI Desktop.

Microsoft Acquires Metanautix

If you’ve missed the announcement from a couple of weeks ago, Microsoft acquired Metanautix – a startup founded by ex-Google engineers who worked on BigQuery (aka Dremel). Technical details are scarce at point. In fact, the Metanautix website doesn’t exist anymore but there are YouTube videos and slides, such as this one. A while back, I wrote about logical data warehouses, which come in different shapes and names, such as software-defined data marts, distributed data, and, what I call, brute-force queries, such as Amazon QuickSight. It looks like that with this acquisition, Microsoft is hoping to make a step in this direction, especially when it comes to Big Data analysis.

From I was able to gather online to connect the pieces, Metanautix Quest uses a SQL-like language to define tables that point to wherever the data resides, such as in HDFS, flat files, or RDBMS. The syntax to define a table might like this:

DEFINE TABLE t AS /path/to/data/*

SELECT TOP(signal1, 100), COUNT(*) FROM t

I believe that the original Google implementation would leave the data on the Google File System (GFS). However, it looks like Metanautix always brings the data into an in-memory columnar store, similar to how Tabular stores the data. When the user sends a query (the query could relate data from multiple stores), a multi-level serving tree algorithm is used to parallelize the query and fetch the data with distributed joins, as described in more details in the “Dremel: Interactive Analysis of WebScale Datasets” whitepaper by Google. According to the whitepaper, this query execution pattern outperforms by far MapReduce queries.

While I was reading about Metanautix, I couldn’t help but ask myself “how is it different than Tabular if it brings the data in?” Yet, from the announcement:

“With Metanautix technology, IT teams can connect a diversity of their company’s information across private and public clouds, without having to go through the costly and complex process of moving data into a centralized system.”

It might that Metanautix is more scalable when it comes to Big Data although I don’t see how this could happen if the data is not in situ. We shall see as details start coming in. “In the coming months, we will have more to share about how we will bring Metanautix technology into the Microsoft data platform, including SQL Server and the Cortana Analytics Suite” One thing is for sure: as with logical data warehouses, Metanautix won’t solve your data integration challenges and it’s not a replacement for DW. From what I can tell, it could help with ad hoc analysis across distributed datasets without having to build analytical models, with all the pros and cons surrounding it.

SSRS Treemap and Sunburst Charts

Years ago, I wrote a blog about how, with some code wizardry, you can create heat maps with Reporting Services. Moving to SSRS 2016, you don’t have to do this anymore thanks to the new Treemap chart type!

Speaking of new charts, SSRS also adds a Sunburst chart.

 

Integrate SSRS 2016 Reports with Power BI

As I said in the “Microsoft Unveils BI Roadmap” blog, SSRS will play an important role of the on-premises BI portal. With this post, I’ll start covering the new SSRS 2016 features. One of the Power BI features announced at the PASS Summit was the forthcoming integration between SSRS 2016 and Power BI that will allow users to pin SSRS report items to Power BI dashboards. You can test pining reports items to Power BI with SQL Server 2016 CTP 3.0 and later. The newly released CTP 3.2 also features the new sleek SSRS portal (we’ve come a long way!). The items you can pin currently include charts, gauge panels, maps, and images. Pinning a report item takes a few clicks:

  1. Open the report and click the Power BI button (a new toolbar button in SSRS 2016). If this is the first time you do this, you’ll be asked to authenticate with Power BI and grant SSRS special permissions it needs for pinning. Then, SSRS switches to a special page mode that shows which items can be pinned.

  1. Click the tile you want to pin.
  2. In the “Pin to Power BI Dashboard” window, select the dashboard you want to pin the report to and the update frequency that specifies how often Power BI will query the report for data changes.

And, the report item is pinned to the dashboard as a regular tile. You can click the tile to navigate back to the SSRS report.

As you can see, Power BI is becoming a focal point for cloud hosting of the main MS report artifacts: Power BI reports, Excel reports, and SSRS reports.

The One and Only Power BI Book

To me, Power BI is the most exciting milestone in the Microsoft BI journey since circa 2005, when Microsoft got serious about BI. Power BI changes the way you gain insights from data; it brings you a cloud-hosted, business intelligence and analytics platform that democratizes and opens BI to everyone.

I’m happy to announce my latest (7th) book – Applied Microsoft Power BI. Currently, the one and only book on Power BI. Some people discouraged me to write this one. After all, trying to cover a product that changes every week is like trying to hit a moving target. However, I believe that the product’s fundamentals won’t change and once you grasp them, you can easily add on knowledge as Power BI evolves over time. Because I had to draw a line somewhere, “Applied Microsoft Power BI” covers all features that were announced at the PASS Summit 2015 and that were released by December 2015.

The book has four parts for each of the four user types: information worker, data analyst, pro, and developer. Information workers will learn how to connect to popular cloud services to derive instant insights, create interactive reports and dashboards, and view them in the browser and on the go. Data analysts will discover how to integrate and transform data from virtually everywhere, and then implement sophisticated self-service models. The book teaches BI and IT pros how to establish a trustworthy environment that promotes collaboration and how to implement Power BI-centric solutions for descriptive, real-time, and predictive analytics. Developers will find how to integrate custom applications with Power BI, embed reports, and implement custom visuals to present effectively any data.

The book is making its way slowly through the retail channels but it should be available on Amazon (in both paper and Kindle formats) and with other retailers before the holidays. I’ll drop a note and update the book page once the book is available for purchase. Meanwhile, visit the book page for more information about the book, source code, and a sample chapter (Chapter 1 “Introduction to Power BI”).

I predict that 2016 will be the year of Power BI and I hope that this book will help you bring your data to life!

Announcing Sparkline Visual

A sparkline is a miniature graph, typically drawn without axes or coordinates. Commonly used on dashboards, sparklines typically visualize trends over time, such as to show profit over the past several years. Although other Microsoft reporting tools, such as Excel and Reporting Services include sparkline elements, Power BI doesn’t have one. That is until now.

I contributed a “classic” sparkline custom visual to the Power BI Visuals Gallery. Once you import and add the sparkline to a report, you bind the sparkline to data using the Data tab of the Visualization pane. For example, you can aggregate a Freight field added the Value area over the CalendarQuarterDesc field added to the Category area. The resulting graph will sho how freight fluctuates over time. You can use any field to group the data, not just a field from the Date table.

The sparkline supports several formatting options. The General section lets you change the line color and width. The default properties are “steelblue” as a color and one pixel for the line width. The Animation section lets you turn on an animation effect that draws the line gradually from left to right (who says that Power BI visuals have be to static?) Although in general I advise against animations and other visual distractors in real-life reports, I wanted to emphasize the point that Power BI visuals can support anything clients-side JavaScript can do. The Duration setting controls how fast the line draws (the default setting is 1,000 milliseconds) and the Delay settings controls the interval between redraws (the default is 3,000 milliseconds). The sparkline supports also a tooltip. When you hover anywhere on the viewport, a tooltip pops up that shows the name of the field added to the Value area and the data point values.

The main limitation of the current implementation is that similar to the Power BI gauge visualization, it doesn’t render over multiple lines, such as for each product category on rows. Preferably, at some point Power BI would support a repeater control similar to the SSRS Tablix region. This would allow nesting the sparkline into other visualizations, such as a table, that will repeat the sparkline for each row. As Power BI stands now, the only way to implement this feature is to draw the visual for each value of the category field, which would require also drawing labels for the row categories. However, it doesn’t repeat on rows, the sparkline could be used to display multiple measures arranged either horizontally or vertically as the screenshot shows. Another limitation related to the previous I’ve just discussed is that it supports only a single field in the Category area and a single field in the Value area. In other words, the sparkline is limited to one measure and one group. Continuing on the list of limitations, the tooltip displays the same information irrespective where you hover on the sparkline viewport as opposed to showing just the value of the data point behind the cursor.

I hope you’ll find my sparkline useful not only for your real-life projects but also for learning how to implement custom visuals.

New Tabular Schema in SQL Server 2016

Those of you who had to generate programmatically Tabular models or make changes to the schema would agree that it was more complicated that it should have been. That’s because Tabular borrowed the schema from Multidimensional so objects had to be defined in different places and in the SSAS MD terminology (dimensions, DSVs, etc). SQL Server 2016 CTP3 brings a new and much more simplified Json-based schema for describing SSAS Tabular models. To see it, you need to configure your workspace database in SQL Server 2016 (1200) compatibility mode. SSMS doesn’t yet support the new schema but you can take a look at Model.bim once you create your project. When you open the source of the Model.bim file you’ll see something completely different than the XML schema you were used to:

{

“name”: “SemanticModel”,

“id”: “SemanticModel”,

“compatibilityLevel”: 1200,

“readWriteMode”: “readWrite”,

“model”: {

“culture”: “en-US”,

“dataSources”: [

{

“name”: “DW”,

“connectionString”: “Provider=SQLNCLI11;Data Source=.;Initial Catalog=AdventureworksDW2016CTP3;Integrated Security=SSPI;Persist Security Info=false”,

“impersonationMode”: “impersonateServiceAccount”,

“annotations”: [

{

“name”: “ConnectionEditUISource”,

“value”: “SqlServer”

},

{

“name”: “PasswordRemoved”

}

]

}

],

“tables”: …

So faster Tabular schema operations in SQL 2016 and much easier model auto-generation. Now all we need is a programmatic way to work with the new schema and this should probably come soon.