At the same time, temporal tables are somewhat more difficult to work with. For example, you must disable system versioning before you alter the table. Here is the recommended approach for altering the schema by the documentation:
BEGIN TRANSACTION ALTER TABLE [dbo].[CompanyLocation] SET (SYSTEM_VERSIONING = OFF); ALTER TABLE [CompanyLocation] ADD Cntr INT IDENTITY (1, 1); ALTER TABLE [dbo].[CompanyLocation] SET ( SYSTEM_VERSIONING = ON (HISTORY_TABLE = [dbo].[CompanyLocationHistory]) ); COMMIT;
However, if you follow these steps, you will be greeted with the following error when you attempt to restore the system versioning in the third step:
Cannot set SYSTEM_VERSIONING to ON when SYSTEM_TIME period is not defined and the LEDGER=ON option is not specified.
Instead, the following works:
ALTER TABLE <system versioned table> SET (SYSTEM_VERSIONING = OFF) -- disable system versioning temporarily -- make schema changes, such as adding new columns ALTER TABLE <system versioned table> ADD PERIOD FOR SYSTEM_TIME (StartDate, EndDate); -- restore time period ALTER TABLE [mulesoft].[Employee] SET (SYSTEM_VERSIONING = ON (HISTORY_TABLE = [mulesoft].[EmployeeHistory])) -- restore system versioning]]>
Presentation: Accelerating your Fabric Data Estate with AI & Copilot
Delivery: In-person
Date: November 4th, 2024
Time: 18:30 – 20:30 ET
Level: Beginner to Intermediate
Food: Pizza and drinks will be provided
Agenda:
18:15-18:30 Registration and networking
18:30-19:00 Organizer and sponsor time (events, Power BI latest, sponsor marketing)
19:00-20:15 Main presentation
20:15-20:30 Q&A
Venue
Improving Office
11675 Rainwater Dr
Suite #100
Alpharetta, GA 30009
Overview: In this presentation, we will explore the groundbreaking AI and Copilot capabilities within Microsoft Fabric, a comprehensive platform designed to enhance productivity and collaboration. By leveraging advanced machine learning algorithms and natural language processing, Microsoft Fabric’s AI/Copilot not only streamlines workflows but also provides intelligent insights and automation, empowering users to achieve more with less effort. Join us as we delve into the features and functionalities that make Microsoft Fabric an indispensable tool for modern enterprises.
Sponsor: CloudStaff.ai
Given the existing technology limitations, you have two implementation choices for implementing subsequent role-playing dimensions: duplicating the dimension table (either in DW or semantic model) or denormalizing the dimension fields into the fact table. The following table presents pros and cons of each option:
Option | Pros | Cons |
Duplicate dimension table in semantic model or DW
|
No or minimum impact on ETL
Minimum maintenance in semantic model All dimension attributes are available |
Metadata complexity and confusion
(potentially mitigated with perspectives that will filter metadata for specific subject area) |
Denormalizing fields from into fact table | Avoid role-playing dimension instances
More intuitive model to business users |
Increased fact table size and memory footprint
Impact on ETL Limited number of dimension attributes Track visited dimension changes as Type 2 with incremental extraction (while it could be Type 1) If applicable, inability to reuse the role-playing dimension for another fact table and do cross-fact table analysis |
So, which approach should you take? The middle path might make sense. If you need only a limited number of fields for the second role-playing dimension, you could add them to the fact table to avoid another dimension and confusion. For example, if you have a DimEmployee dimension and you need a second instance for the person making the changes to the fact table, you can add the administrator’s full name to the fact table assuming you need only this field from DimEmployee.
By contrast, if you need most of the fields in the role-playing instances, then cloning might make more sense. For example, analyzing fact data by shipped date or due date that requires the established hierarchies in DimDate, could be addressed by cloning DimDate. Then to avoid confusion, consider using Tabular Editor to create perspectives for each subject area where each perspective includes only the role-playing dimensions applicable to that subject area.
Yet, a narrow-case third option exists when you only need role-playing measures, such as SalesAmountByShipDate and SalesAmountByDueDate. This scenario can be addressed by forcing DAX measures to “travel” the inactive relationship by using the USERELATIONSHIP function.
]]>
I have to say that I was impressed with LLM. I used the excellent Ric Zhou’s Text2SQL sample as a starting point inside Visual Studio Code.
The sample uses the Python streamlit framework to create a web app that submits natural questions to Azure OpenAI. I was amazed how simple the LLM input was. Given that it’s trained with many popular languages, including SQL, all you have to do is provide some context, database schema (generated in a simple format by a provided tool), and a few prompts:
[{'role': 'system', 'content': 'You are smart SQL expert who can do text to SQL, following is the Azure SQL database data model <database schema>}, {'role': 'user', 'content': 'What are the three best selling cities for the "AWC Logo Cap" product?'}, {'role': 'assistant', 'content': 'SELECT TOP 3 A.City, sum(SOD.LineTotal) AS TotalSales \\nFROM [SalesLT].[SalesOrderDe...BY A.City \\nORDER BY TotalSales DESC; \\n'}, {'role': 'user', 'content': natural question here'}]
Let’s drill into these prompts.
You can find more Text2SQL implementation details in my post “LLM Adventures: Text2SQL”.
As a Microsoft BI practitioner, the next natural stop was Text2DAX. But wait, we have a Microsoft Fabric Copilot already for this, right? Yes, but what happens when you click the magic button in PBI Desktop? You are greeted that you need to purchase F64 or larger capacity. It’s a shame that Microsoft has decided that AI should be a super-premium feature. Given this horrible predicament, what would an innovative developer strapped for cash do? Create their own copilot of course!
Building upon the previous sample, this is remarkably simple. First, I obtained the model schema using Analysis Services Data Management Views (DMVs). Second, I changed slightly the prompt, along these lines:
“As a DAX expert, examine a Power BI data model with the following table definitions where TableName lists the tables and ColumnName lists the columns, and provide DAX query that can answer user questions based on the data model, you will think step by step throughout and return DAX statement directly without any additional explanation.”
You can find more Text2DAX implementation details in my post “LLM Adventures: Text2DAX”.
In summary, it appears that LLM can effectively assist us in writing code. The emphasis is on assist because I view the LLM role as a second set of eyes. Hey, what do you think about this problem I’m trying to solve here? LLM doesn’t absolve us from doing our homework and learning the fundamentals, nor it can compensate for improper design. While LLM might not always generate the optimum code and might sometimes fabricate, it can definitely assist you in creating business calculations, generating test queries, and learning along the way.
BTW, you can use any of the publicly available LLM apps, such as Copilot, ChatGPT, Google Gemini or Perplexity (you don’t need the sample app I’ve demonstrated) for Text2SQL and Text2DAX and probably you will obtain similar results if you give it the right prompts. I took this approach because I was interested in automating the process for business users.
]]>Not sure who Microsoft hired to cleanse copilots, but apparently they went overboard. Here is a Windows Copilot jerk-knee response that I got to a prompt asking when to open with 2 No Trump in contract bridge.
I see, Generative AI refuse to generate since “trump” is a taboo topic nowadays. AI dummy!
]]>Understanding the Tests
My starting hypothesis was that Direct Lake on-demand loading will utilize memory much more efficiently for interactive operations, such as Power BI report execution. This is a bonus to the fact that data Direct Lake models don’t require refresh. Eliminating refresh could save tremendous amount of memory to start with, even if you apply advanced techniques such as incremental refresh or hybrid tables to models with imported data. Therefore, the tests that follow focus on memory utilization with interactive operations.
To test my hypothesis, I imported the first three months for year 2016 of the NY yellow taxi Azure open dataset to a lakehouse backed up by a Fabric F2 capacity. The resulted in 34.5 million rows distributed across several Delta Parquet files. I limited the data to three months because F2 ran out of memory around the 50 million rows mark with the error “This operation was canceled because there wasn’t enough memory to finish running it. Either reduce the memory footprint of your dataset by doing things such as limiting the amount of imported data, or if using Power BI Premium, increase the memory of the Premium capacity where this dataset is hosted. More details: consumed memory 2851 MB, memory limit 2851 MB, database size before command execution 220 MB”
Descriptive enough and in line with the F2 memory limit of maximum 3 GB per semantic model. I used Power BI desktop to import all that data into a YellowTaxiImported semantic model, which I published to Power BI Service and configured for large storage format. Then, I created online a second YellowTaxiDirectLake semantic model configured for Direct Lake storage mapped directly to the data in the lakehouse. I went back to Power BI desktop to whip up a few analytical (aggregate) queries and a few detail-level queries. Finally, I ran a few tests using DAX Studio.
Analyzing Import Mode
Even after a capacity restart, the YellowTaxiImported model immediately reported 1.4 GB of memory. My conclusion was that that the primary focus of Power BI Premium on-demand loading that was introduced a while back was to speed the first query after the model was evicted from memory. Indeed, I saw that many segments were memory resident and many weren’t, but using queries to touch the non-resident column didn’t increase the memory footprint. The following table lists the query execution times with “Clear On Run” enabled in DAX Studio (to avoid skewing due to cached query data).
Naturally, as the queriers get more detailed, the slower they get because VertiPaq is a columnar database. However, the important observation is that the memory footprint remains constant. Please note that Fabric allocates additional memory to execute the queries, so the memory footprint should grow up as the report load increases.
Query | Duration (ms) |
//Analytical query 1
EVALUATE |
124 |
//Analytical query 2
DEFINE EVALUATE |
148 |
//Analytical query 3
DEFINE VAR __SQDS0BodyLimited = VAR __DS0Core = VAR __DS0PrimaryWindowed = EVALUATE ORDER BY |
114 |
//Analytical query 4
DEFINE VAR __SQDS0BodyLimited = VAR __DS0Core = VAR __DS0BodyLimited = ORDER BY |
80 |
//Detail query 1
DEFINE VAR __DS0FilterTable2 = VAR __DS0Core = VAR __DS0PrimaryWindowed = EVALUATE ORDER BY |
844 |
//Detail query 2
DEFINE VAR __DS0FilterTable2 = VAR __DS0Core = VAR __DS0PrimaryWindowed = EVALUATE ORDER BY |
860 |
//Detail query 3
DEFINE VAR __DS0FilterTable2 = VAR __DS0Core = VAR __DS0PrimaryWindowed = EVALUATE ORDER BY |
1,213 |
//Detail query 4 (All Columns)
DEFINE VAR __DS0FilterTable2 = VAR __ValueFilterDM1 = VAR __DS0Core = VAR __DS0PrimaryWindowed = EVALUATE ORDER BY |
4,240 |
Analyzing Direct Lake
After another restart, the resident memory footprint of the YellowTaxiDirectLake model was only 22.4 KB! Indeed, the $System.DISCOVER_STORAGE_TABLE_COLUMN_SEGMENTS DMV showed that only system-generated RowNumber columns were memory resident.
For each query, I recorded two runs to understand how much time is spent in on-demand loading of columns into memory. The Import Mode column was added for convenience to compare the second run duration with the corresponding query duration from the Import Mode tests. Finally, the Model Resident Memory column records the memory footprint of the Direct Lake model.
Query | First Run (ms) |
Second Run (ms) | Import Mode
(ms) |
Model Resident Memory (MB) |
//Analytical query 1 | 79 | 75 | 124 | 14 |
//Analytical query 2 | 79 | 76 | 148 | 14.3 |
//Analytical query 3 | 382 | 133 | 114 | 68.1 |
//Analytical query 4 | 209 | 130 | 80 | 68.13 |
//Detail query 1 | 7,763 | 1,023 | 844 | 669.13 |
//Detail query 2 | 1,484 | 1,453 | 860 | 670.53 |
//Detail query 3 | 1,881 | 1,463 | 1,213 | 670.6 |
//Detail query 4 | 9,663 | 3,668 | 4,240 | 1,270 |
Conclusion
To sum up this long post, the following observations can be made:
]]>Therefore, if Direct Lake is an option for you, it should be at the forefront of your efforts to combat out-of-memory errors with large datasets. On the downside, more than likely you’ll have to implement ETL processes to synchronize your data warehouse to a Fabric lakehouse, unless your data is in Fabric to start with, or you use Fabric database mirroring for the currently supported data sources (Azure SQL DB, Cosmos, and Snowflake). I’m not counting the data synchronization time as a downside because it could supersede the time you currently spend in model refresh.
SKU | Max memory (GB)1, 2 | Max concurrent DirectQuery connections (per semantic model)1 | Max DirectQuery parallelism3 | Live connection (per second)1 | Max memory per query (GB)1 |
Model refresh parallelism | Direct Lake rows per table (in millions)1, 4 | Max Direct Lake model size on OneLake (GB)1, 4 |
F2 | 3 | 5 | 1 | 2 | 1 | 1 | 300 | 10 |
F4 | 3 | 5 | 1 | 2 | 1 | 2 | 300 | 10 |
F8 | 3 | 10 | 1 | 3.75 | 1 | 5 | 300 | 10 |
F16 | 5 | 10 | 1 | 7.5 | 2 | 10 | 300 | 20 |
F32 | 10 | 10 | 1 | 15 | 5 | 20 | 300 | 40 |
F64 | 25 | 50 | 4-8 | 30 | 10 | 40 | 1,500 | Unlimited |
F128 | 50 | 75 | 6-12 | 60 | 10 | 80 | 3,000 | Unlimited |
F256 | 100 | 100 | 8-16 | 120 | 10 | 160 | 6,000 | Unlimited |
F512 | 200 | 200 | 10-20 | 240 | 20 | 320 | 12,000 | Unlimited |
F1024 | 400 | 200 | 12-24 | 480 | 40 | 640 | 24,000 | Unlimited |
F2048 | 400 | 200 | 960 | 40 | 1,280 | 24,000 | Unlimited | |
The same page lists another important table that shows the background CPU cores assigned to a capacity. Although bursting, overages, smoothing, and throttling make Fabric capacity compute resources a whole lot more difficult to figure out, think of your capacity as a VM that has that many cores for backend loads, including loads from interactive operations, such as report queries, and loads from background operations, such as dataset refreshes. Not sure what it shows N/A for F2 and F4. If memory serves me right, I’ve previously seen 0.25 cores for F2 and 0.5 cores for F4.
SKU | Capacity Units (CU) | Power BI SKU | Power BI v-cores |
F2 | 2 | N/A | N/A |
F4 | 4 | N/A | N/A |
F8 | 8 | EM1/A1 | 1 |
F16 | 16 | EM2/A2 | 2 |
F32 | 32 | EM3/A3 | 4 |
F64 | 64 | P1/A4 | 8 |
F128 | 128 | P2/A5 | 16 |
F256 | 256 | P3/A6 | 32 |
F5121 | 512 | P4/A7 | 64 |
F10241 | 1,024 | P5/A8 | 128 |
F20481 | 2,048 | N/A | N/A |
]]>
Presentation: Create Code Copilots with Large Language Models
Delivery: In-person
Date: September 3rd, 2024
Time: 18:30 – 20:30 ET
Level: Beginner to Intermediate
Food: Pizza and drinks will be provided
Agenda:
18:15-18:30 Registration and networking
18:30-19:00 Organizer and sponsor time (events, Power BI latest, sponsor marketing)
19:00-20:15 Main presentation
20:15-20:30 Q&A
Venue
Improving Office
11675 Rainwater Dr
Suite #100
Alpharetta, GA 30009
Overview: Resistance is futile! Instead of fearing that AI will take over our jobs, embrace it and apply it to outsource mundane work and create a new class of applications that were not possible before. In this session, I’ll introduce you through the fascinating world of Large Language Models (LLMs) and one of their practical applications in creating Text2SQL and Text2DAX copilots. I’ll demonstrate how LLMs open new opportunities for intelligent exploration. As an optional challenge, bring your laptop, download the code from my website (use the download link below), and follow along using your favorite AI chat, such as ChatGPT, which is what I’ll use for the demos, Microsoft Copilot, Meta AI, Google Gemini, or Perplexity.ai. You’ll also discover how you can automate LLM-powered copilots using Python and Azure OpenAI.
Code download link: Create Code Copilots with Large Language Models
Speaker: Teo Lachev is a BI consultant, author, and mentor. Through his Atlanta-based company Prologika (https://prologika.com) he designs and implements innovative solutions that bring tremendous value to his clients and help them make sense of data. Teo has authored and co-authored many books, and he has been leading the Atlanta Microsoft Business Intelligence group since he founded it in 2010. Microsoft has recognized Teo’s contributions to the community by awarding him the prestigious Microsoft Most Valuable Professional (MVP) Data Platform status for 15 years. Microsoft selected Teo as one of only 30 FastTrack Solution Architects for Power Platform worldwide.
Sponsor: Prologika (https://prologika.com)
Since its beginning, Power BI Pro per-user licensing (and later Premium Per User (PPU) licensing) has been very attractive. Many organizations with a limited number of report users flocked to Power BI to save cost. However, organizations with more BI consumers gravitated toward premium licensing where they could have unlimited number of report readers against a fixed monthly fee starting at listed price of $5,000/mo for P1. Sounds like a great deal, right?
I must admit that I detest the premium licensing model because it boxes into certain resource constraints, such as 8 backend cores and 25 GB RAM for P1. There are no custom configurations to let you balance between compute and memory needs. And while there is an auto-scale compute model, it’s very coarse. The memory constraints are especially problematic given that that imported models are memory resident and require more than twice the memory for full refresh. From the outside, these memory constraints seem artificially low to force clients into perpetual upgrades. The new Fabric F capacities that supersede the P plans are even more expensive, justifying the price increase with the added flexibility to pause the capacity which is often impractical.
It looks to me that the premium licensing is pretty good deal for Microsoft. Outgrown 25 GB of RAM in P1? Time to shelve another 5K per month for 25 GB more even if you don’t need more compute power. Meanwhile, the price of 32GB of RAM is less than $100 and falling.
So, what should you do if you are strapped for cash? Consider evaluating and adopting one or more of the following techniques, including:
It will be great if at some point Power BI introduces customized capacities. Even better, how about auto-scaling where the capacity resources scale up and down on demand within minutes, such as adding more memory during refresh and reducing the memory when the refresh is over?
]]>
Presentation: Elevate Program Management with Power BI & DevOps
Delivery: In-person
Date: August 5, 2024
Time: 18:30 – 20:30 ET
Level: Intermediate
Food: Pizza and drinks will be provided
Agenda:
18:15-18:30 Registration and networking
18:30-19:00 Organizer and sponsor time (events, Power BI latest, sponsor marketing)
19:00-20:15 Main presentation
20:15-20:30 Q&A
Venue
Improving Office
11675 Rainwater Dr
Suite #100
Alpharetta, GA 30009
Overview: Have you ever opened Azure DevOps and felt overwhelmed by the vast sea of program management options? In large organizations, tracking progress across disparate projects and work items can be challenging. In this session, find out how to bring Azure DevOps data to life by creating data models and interactive reports in Power BI. Sleek Power BI visuals make even the most technical DevOps content both accessible to executives and actionable for project managers.
Speaker: Elayne Jones and Matt Kim are both Solutions Architects at Coca-Cola Bottlers Sales and Services. Elayne and Matt specialize in developing solutions that drive efficiency within organizations by utilizing the full set of Power Platform technologies. Elayne and Matt work together on a team focusing on designing and implementing automated solutions to enhance both internal and external stakeholders’ user experiences and to enforce consistency in reporting data.
Sponsor: Cloudstaff.ai