Direct Lake Composite Models

I’ve mentioned previously in the “A Couple of Direct Lake Gotchas” post that unlike Power BI reports, Excel surprised me by not showing user-defined hierarchies in a Direct Lake model. Direct Lake comes with other gotchas, such as not supporting DAX calculated columns and SQL views. I normally don’t use calculated columns, but they can come in handy, such as for flatting a parent-child hierarchies outside ETL. And I like SQL views as an insurance policy for making quick transforms or filters on top of loaded tables to avoid ETL changes.

Recently, Microsoft introduced composite Direct Lake models which I demonstrated as a part of a pilot project, mainly to preserve the Excel report experience for financial users.

Direct Lake Only vs Composite Direct Lake

I view the composite Direct Lake models as the best of both worlds and this table summarizes their characteristics.

Direct Lake Only Composite (Direct Lake + Import)
Public Preview
Storage mode Direct Lake Some tables, such as dimensions, in import mode, others, such as fact tables, in Direct Lake
Tooling Web modeling, Power BI Desktop Web modeling only (limited support in PBI Desktop)
Refresh required No (Fabric monitors the Delta log) Imported tables must be refreshed, such as overnight for dimension changes
Memory consumption Used columns in reports are paged in and out Refresh requires at least twice the memory of imported objects
SQL Views No Yes
Calculated columns No Yes, such as PATH and PATHITEM to flatten parent-child hierarchies outside ETL
User-defined hierarchies Power BI reports Power BI, Excel, and Anaplan
Power Query No Yes

Therefore, composite Direct Lake models could be a good compromise between real-time BI and flexibility. For example, now you can implement the following configuration:
1. Dimensions in Import mode and refresh them overnight since they probably don’t change frequently anyway.
2. Large fact tables or tables requiring real-time BI in Direct Lake without refresh.

Lessons Learned

If composite models sound appealing, you might be eager to convert an existing Direct Lake model to composite. Here are some issues/gotchas that I ran into doing so:

  1. The web modeling experience (currently, the only way to add imported tables using Microsoft tooling) would show the Get Data buttons disabled. After some reverse-engineering of a brand new model, I fixed it my changing the connection expression in the *.bim file to use onelake (previously, it was pointing to a Fabric warehouse).
"expression": [
"let",
" Source = AzureStorage.DataLake(\"https://onelake.dfs.fabric.microsoft.com/44744d84-...\", [HierarchicalNavigation=true])",
  1. If you’re switching many tables from Direct Lake, you can automate this in two ways:
import sempy_labs as sl
from sempy_labs import migration
migration.migrate_direct_lake_to_import(
  dataset="Your_Direct_Lake_Model_Name",
  workspace="Your_Workspace_Name",
  mode="import" # or "directquery"
)
  • C# script in the Tabular Editor
  1. In my case, since there weren’t that many tables, I converted the dimension table partitions manually to “M” partitions, as in this example (change bolded text for each table):
    "partitions": [
{
"name": "Account",
"mode": "import",
"source": {
"expression": [
"let",
" Source = Sql.Database(\"<guid>.datawarehouse.fabric.microsoft.com\", \"<warehouse name>\"),",
" #\"Navigation 1\" = Source{[Schema = \"dbo\", Item = \"DimAccount\"]}[Data]",
"in",
" #\"Navigation 1\""
],
"type": "m"
}
} ]
  1. After switching a partition from Direct Lake to Import in PBI Desktop project and synchronizing to the connected published model, Fabric rejected the change complaining that existing DirectLake table can’t be switched to imported storage. As a workaround, I dropped the connected model.
  2. Being in public review, Composite Direct Lake is rough around the edges. I’ve got various complaints about credentials missing which I fixed in the dataset settings.
  3. Although the documentation says that web modeling is the only tooling experience, Power BI Desktop worked for me as with the Direct Lake only counterpart. However, currently Power Query and Get Data (unless you add the tables directly in *.bim file) is only available on the web.

A “Limited” Performance Note

I know everyone is interested in performance. I did some limited performance tests by tracing a massive query against equivalent Direct Lake Only and Composite Direct Lake models. On a cold cache, Composite outperformed Direct Lake Only by some 20%. On a warm cache, surprisingly I saw the reverse, Direct Lake Only outperforming Composite five to six times. Please don’t take this verbatim. More than likely, your results will vary. For example, in that blog I said that I saw much better performance with SWITCH…CASE in Imported vs Direct Lake. Test!

 

Atlanta Microsoft BI Group Meeting on March 2nd (Your First Steps in Microsoft Fabric Using Just SQL)

Atlanta BI fans, please join us in person for our next meeting on Monday, March 2nd at 18:30 ET. Shabnam Watson will show you how you can apply your SQL skills in Microsoft Fabric. And your humble correspondent will walk you through some of the latest Power BI and Fabric enhancements. I will sponsor the meeting. For more details and sign up, visit our group page.

Delivery: In-person
Level: Beginner/Intermediate
Food: Pizza and drinks will be provided

Agenda:
18:15-18:30 Registration and networking
18:30-19:00 Organizer and sponsor time (news, Power BI latest, sponsor marketing)
19:00-20:15 Main presentation
20:15-20:30 Q&A

Overview: New to Microsoft Fabric? Don’t worry—you already know more than you think. In this beginner-friendly session, we’ll explore how your existing SQL skills translate directly into Fabric without needing to learn Spark, Python, or unfamiliar engineering tools. You’ll see how SQL can be applied across Fabric items to explore, shape, and analyze data with confidence. If you’re just beginning your Fabric journey, this session offers a simple, approachable path to success using the skills you already have.

Speaker: Shabnam is a business intelligence consultant and owner of ABI Cube, a company that specializes in delivering data solutions using the Microsoft Data Platform. She has over 20 years of experience and is recognized as a Microsoft Data Platform MVP for her technical excellence and community involvement. She is passionate about helping organizations harness the power of data to drive insights and innovation. She has a deep expertise in Microsoft Analysis Services, Power BI, Azure Synapse Analytics, and Microsoft Fabric. She is also a speaker, blogger, and organizer for SQL Saturday Atlanta – BI version, where she shares her knowledge and best practices with the data community.

Sponsor: Prologika (https://prologika.com) helps organizations of all sizes to make sense of data by delivering tailored BI solutions that drive actionable insights and maximize ROI. Your BI project will be your best investment, we guarantee it!

PowerBILogo

A Couple of Direct Lake Gotchas

I’m helping an enterprise client modernize their data analytics estate. As a part of this exercise, a SSAS Multidimensional financial cube must be converted to a Power BI semantic model. The challenge is that business users ask for almost real-time BI during the forecasting period, where a change in the source forecasting system must be quickly propagated to the reporting the layer, so the users don’t sit around waiting to analyze the impact. An important part of this architecture is the Fabric Direct Lake storage to eliminate the refresh latency, but it came up with a couple of gotchas.

Performance issues with calculated accounts

Financial MD cubes are notoriously difficult to convert to Tabular/Power BI because of advanced features that aren’t supported in the new world, such as Account Intelligence, scope assignments, parent-child hierarchies, and calculated dimension members. The latter presented a performance challenge. Consider the following MDX construct:

CREATE MEMBER CURRENTCUBE.[Account].[Accounts].&[1].[Calculations List].[ROI %]
AS IIF([Account].[Accounts].&[1].[Calculations List].[Average Invested Capital] = 0, NULL, ..,
FORMAT_STRING = "#,##0.0 %;-#,##0.0 %",

This construct adds an expression-based account as though the account is physically present in the chart of accounts. MD evaluates the MDX expression only for that account.


No such a construct exists in Tabular. To provide a similar reporting experience, I attempted to overwrite the Value measure conditionally based on the “current” account, such as:

VAR _Level03 = SELECTEDVALUE ('Account'[Level 03]) 
RETURN 
IF (_Level03 <> "Calculations List", 
       [Value],
       VAR _Level04 = SELECTEDVALUE ('Account'[Level 04])
       RETURN
            SWITCH (
               Level04,
               "ROI %", [ROI],
               "Economic Profit", [Economic Profit],
…

However, no matter what I tried, the report performance got a big dent (from milliseconds to 10+ seconds) even when the Calculations List account was excluded. Interestingly, report performance in Direct Lake fared 2-3 times worse than an equivalent Power BI imported model.

So, we had to scrap this approach in favor of one of these workarounds:

  1. Pre-calculating the calculated accounts values (materializing)
    1. Pros: same reporting behavior as MD, faster performance compared to MDX expressions
    2. Cons: effort shifted to ETL, potentially impacting real-time forecasting if calculations must be recomputed with each change.
  2. Separate DAX measures
    1. Pros: formulas applied at runtime as MD, no impact on ETL
    2. Cons: different report experience

Excel dropping user-defined hierarchies

Excel never fails to disappoint me. Sad, considering its potential as an alternative reporting client, especially for financial users.

This time Excel pivots decided not to show user-defined hierarchies, which turns out to be a document limitation for DirectQuery and Direct Lake. Microsoft provides no explanation and I’m sure the Excel team has no plans to fix it, as well as to finally embrace DAX and Power BI semantic models.

Luckily, the client uses a third-party Excel-based tool, which provides better report experience and supports user-defined hierarchies. If the Excel limitation becomes an issue, Fabric Direct Lake is expected soon to support composite models. This will let you implement models with hybrid storage, such as importing dimensions, which don’t change frequently, but leave fact tables in Direct Lake. Luckily, Excel supports user-defined hierarchies with imported tables.

Atlanta Microsoft BI Group Meeting on February 2nd (Power BI Translytical Taskflows)

Atlanta BI fans, please join us in person for our next meeting on Monday, February 2nd at 18:30 ET. Sukhwant Kaur (Product Manager at Microsoft) will show you how supercharge your Power BI reports with translytical taskflows. And your humble correspondent will walk you through some of the latest Power BI and Fabric enhancements. CloudStaff.au will sponsor the meeting. For more details and sign up, visit our group page.

Delivery: In-person
Level: Beginner/Intermediate
Food: Pizza and drinks will be provided

Agenda:
18:15-18:30 Registration and networking
18:30-19:00 Organizer and sponsor time (news, Power BI latest, sponsor marketing)
19:00-20:15 Main presentation
20:15-20:30 Q&A

Overview: Join us for an engaging session exploring how to build powerful translytical applications using Power BI, Functions, and SQL Database within Microsoft Fabric. We’ll discuss best practices for integrating analytics and transactional workloads, demonstrate real-world use cases, and provide actionable tips for leveraging Fabric’s unified platform. This talk is ideal for data professionals interested in bridging analytics and operations for enhanced business value.

Speaker: Sukhwant has served as a Product Manager at Microsoft for the past few development cycles. During this time, she’s focused on the entire product management lifecycle, from working with development teams and user experience to collaborating with cross-functional teams to drive customer satisfaction in ensuring our products not only meet but exceed customer expectations.

Sponsor: At CloudStaff.ai we’re making work MORE. HUMAN. We believe in the power of technology to enhance human potential, not replace it. Our innovative AI and automation solutions are designed to make work easier, more efficient, and more meaningful. We help businesses of all sizes streamline their operations, boost productivity, and solve real-world challenges. Our approach combines cutting-edge technology with a deep understanding of human needs, creating solutions that work the way people do! https://cloudstaff.ai

PowerBILogo