Blog – Page 4 – Prologika

Notes on Fabric F2 Performance: Report Load

December 25, 2023 / No Comments »

What a better way to spend a lazy holiday afternoon than to do more Fabric performance testing? In my previous post, I shared my results from a single-threaded ETL load test to gauge the F2 ingest performance and F2 did pretty well (or at least outperformed Azure SQL DB). Will F2 hold as parallelism increases? Throughput testing is especially important for report loads because parallel tasks can run within a report, such as visuals executing DAX queries in parallel, and across reports, such as when concurrent report requests overlap. I used the artifacts are included in the "Microsoft Dashboard in a Day" for this test and load tested only the first report page. The Sales fact table in the semantic model has over seven million rows so it represents a good size dataset. Naturally, the more involved the report is and the more data the semantic model has, the more...
Notes on Fabric F2 Performance: Warehouse ETL

December 23, 2023 / No Comments »

As inspired by Amir Netz's encouragement to partners to test the Fabric F2 capacity performance, I got on a quest to test what it would do to ETL loads for Fabric Warehouse. I must admit that I was skeptical that a quarter of a core would take a warehouse off the ground, but as usual, life proved me wrong and "wrong" is a big understatement of what happened. After provisioning a Fabric F2 capacity and a warehouse, I settled on the Retail Data Model for World Wide Importers sample star schema dataset consisting of five dimension tables and one fact table. In terms of performance, I was mostly interested in how long it would take for the ADF copy activity to insert all the data (50 million rows) in the fact table. Granted, it's a limited test but enough to rule out the technology for real-life projects. Then, I compared...
A Case for Microsoft Fabric: Dynamics 365 Analytics

December 7, 2023 / No Comments »

I might have identified at last a good case for Microsoft Fabric, but I'll be in a better position to confirm after a POC with larger datasets. Dynamics Online, aka Dynamics 365, epitomizes the customer's struggle to export their data hosted in SaaS cloud offerings for analytics or other purposes. Since unfortunately Microsoft doesn't provide direct access to the Dynamics native storage (Azure SQL Database), which often could be the simplest and fastest solution, Dynamics has "compensated" throughout the years by introducing and sunsetting various options: Option Pros Cons OData API Original interface Slow, no query folding, currently deprecated Data Export Service Export to Azure SQL Database Deprecated TDS endpoint Real-time direct access to Dynamics Azure SQL Database Not designed for massive extracts; slow Dataverse API Currently recommended by Microsoft As far as I can tell, no REST operations (see below); questionable performance Dataverse connector Same as Dataverse API Same...
Atlanta Microsoft BI Group Meeting on December 4th (Financial Modelling in Microsoft Fabric)

November 28, 2023 / No Comments »

Atlanta BI fans, please join us for the next meeting on Monday, December 4th, at 6:30 PM ET. William Rodriguez will review the key Fabric components whilst walking through a full Financial deployment: data engineering with a tiered medallion architecture, data science with forecasting and other advanced analytics, and data analysis with Power BI. Your humble correspondent will help you catch up on Microsoft BI latest. For more details and sign up, visit our group page. PLEASE NOTE A CHANGE TO OUR MEETING POLICY. WE HAVE DISCONTINUED ONLINE MEETINGS VIA TEAMS. THIS GROUP MEETS ONLY IN PERSON. WE WON’T RECORD MEETINGS ANYMORE. THEREFORE, AS DURING THE PRE-PANDEMIC TIMES, PLEASE RSVP AND ATTEND IN PERSON IF YOU ARE INTERESTED IN THIS MEETING. Presentation: Financial Modelling in Microsoft Fabric Delivery: In-person Date: December 4 Time: 18:30 – 20:30 ET Level: Intermediate Food: TBD Agenda: 18:15-18:30 Registration and networking 18:30-19:00 Organizer and...
Atlanta Microsoft BI Group Meeting on November 6th (Exploring Bravo for Power BI)

October 31, 2023 / No Comments »

Atlanta BI fans, please join us for the next meeting on Monday, November 6th, at 6:30 PM ET. Macro Russo will introduce us to the community tool Bravo for Power BI. Your humble correspondent will help you catch up on Microsoft BI latest. I'll sponsor the event. For more details and sign up, visit our group page. PLEASE NOTE A CHANGE TO OUR MEETING POLICY. WE HAVE DISCONTINUED ONLINE MEETINGS VIA TEAMS. THIS GROUP MEETS ONLY IN PERSON. WE WON’T RECORD MEETINGS ANYMORE. THEREFORE, AS DURING THE PRE-PANDEMIC TIMES, PLEASE RSVP AND ATTEND IN PERSON IF YOU ARE INTERESTED IN THIS MEETING. Presentation: Exploring Bravo for Power BI Delivery: In-person Date: November 6 Time: 18:30 – 20:30 ET Level: Beginner Food: Pizza and drinks will be provided Agenda: 18:15-18:30 Registration and networking 18:30-19:00 Organizer and sponsor time (events, Power BI latest, sponsor marketing) 19:00-20:15 Main presentation 20:15-20:30 Q&A Venue Improving...
ADF Cross Environment Schema Compare Tool

October 24, 2023 / No Comments »

Almost every BI project requires at least two environments, such as DEV and PROD. Unless I've missed something, one Azure Data Factory caveat irrespective of the code promotion path (manual or automated with Azure DevOps or Git CI/CD) is that it uses ARM template deployment where the template contains all the artifacts. The equivalent for database schema deployment would be to script the entire database schema which makes me nervous. Not to mention that ARM templates have their own gotchas, such as the one I discussed here when shared self-hosted integration runtimes are used. So, I got on a quest to find an ADF schema comparison tool for a quick and dirty way to identify ADF code discrepancies between two environments and promote changes manually. Ideally, the tool would be something like SQL Server Database projects in Visual Studio. However, I wasn't able to find such a schema comparison tool...
Deactivating ADF Activities

October 18, 2023 / No Comments »

Lo and behold, Azure Data Factory now supports deactivating activities as a preview feature. It took only seven years… You can deactivate/activate activity by right-clicking it in the canvas or from its General settings. You can also configure how the subsequent activities should treat a deactivated activity, with Succeeded being the most common outcome.
When All You Have is a Hammer… (Dbt Tool for ETL)

October 16, 2023 / No Comments »

Someone asked the other day for my opinion about the open-source dbt tool for ETL. I hadn't heard about it. Next thing I've noticed was that Fabric Warehouse added support for it, so I got inspired to take a first look. Seems like an ELT-oriented tool. Good, I'm a big fan of the ELT pattern whose virtues I extolled many times here. But a Python-based tool that requires writing templates in a dev environment, such as Visual Studio Code? Yuck! So, what's my first impression? Same thoughts as when I see developers use another generic programming language, such as C#, for ETL. You can do it but why? For years, I've been trying to get developers out of custom coding for ETL to low-code ETL specialized tools, such as SSIS and ADF. Just because you studied Python in college, should you use Python for everything? I guess the open-source custom...
Atlanta Microsoft BI Group Meeting on October 2nd (30 Power BI Tips and Tricks)

September 26, 2023 / No Comments »

Atlanta BI fans, please join us for the next meeting on Monday, October 2nd, at 6:30 PM ET. Michael Hewitt will share 30 tips to make you more productive with everything Power BI. Your humble correspondent will help you catch up on Microsoft BI latest. For more details and sign up, visit our group page. PLEASE NOTE A CHANGE TO OUR MEETING POLICY. WE HAVE DISCONTINUED ONLINE MEETINGS VIA TEAMS. THIS GROUP MEETS ONLY IN PERSON. WE WON’T RECORD MEETINGS ANYMORE. THEREFORE, AS DURING THE PRE-PANDEMIC TIMES, PLEASE RSVP AND ATTEND IN PERSON IF YOU ARE INTERESTED IN THIS MEETING. Presentation: 30 Power BI Tips and Tricks Delivery: In-person Date: October 2nd Time: 18:30 – 20:30 ET Level: Beginner to Intermediate Food: Sponsor wanted Agenda: 18:15-18:30 Registration and networking 18:30-19:00 Organizer and sponsor time (events, Power BI latest, sponsor marketing) 19:00-20:15 Main presentation 20:15-20:30 Q&A Venue Improving Office 11675 Rainwater...
Hiking the Precipice Trail

September 22, 2023 / No Comments »

One of trails my wife and I hiked today in the Acadia National Park was the Precipice Trail. This trail is considered one of the hardest trails in the United States. In fact, "hiking" is a misnomer as most of the ascent involves vertical rock climbing and pulling yourself up on iron rangs. Slipping and falling can surely result in a serious injury or death as there are no safety nets and there were such incidents in the past. However, if you put yourself in the right mindset and prepare, you probably won't find it that hard. In fact, we've found the descent more challenging than the ascent which we thought was exhilarating. And the Gulf of Main views were breathtaking! How many ascents and views do we miss in life by not taking moderate risks?