Microsoft Purview Data Governance - New Catalog Experience

1. The history of Microsoft Purview

Image

Microsoft Purview is a complete Data Governance solution that has evolved significantly since its inception. It represents Microsoft’s commitment to providing robust solutions for data management, security, and compliance in an increasingly data-driven world.

As data challenges evolved, so did Microsoft Purview, and to better understand its current capabilities, it’s important to look back at how it all started.

Origins and Early Developments

The journey of Microsoft Purview began with recognizing the growing need for effective Data Governance solutions in the cloud. As organizations amassed vast amounts of data, the challenges of managing, securing, and deriving value from this data became more pronounced. Microsoft responded to these challenges by developing a suite of tools to improve Data Governance and compliance.

Introduction of Azure Purview

In December 2020, Microsoft introduced Azure Purview, a unified Data Governance service designed to help organizations manage and govern their on-premises, multi-cloud, and software-as-a-service (SaaS) data. Azure Purview provided a comprehensive set of features, including data discovery, data classification, and data lineage tracking. These capabilities gave organizations a holistic view of their data landscape, ensuring better data management and compliance. However, consumer expectations also encompassed data quality and master data management.

Expansion and Rebranding

As the demand for Data Governance solutions continued to grow, Microsoft expanded the capabilities of Azure Purview. In April 2022, Microsoft rebranded Azure Purview to Microsoft Purview, reflecting its broader scope and enhanced features. The rebranding also signified Microsoft’s commitment to integrating Data Governance more deeply into its suite of cloud services. 

Integration with Microsoft 365 Compliance

One of the significant milestones in the evolution of Microsoft Purview was its integration with Microsoft 365 Compliance. This integration brought together Data Governance, Security and Compliance capabilities, providing organizations with a unified platform to manage their data and ensure regulatory compliance. The combined solution offered advanced features such as information protection, risk management, and compliance reporting.

Introduction of New Features

Microsoft Purview continues to evolve, with new features regularly introduced to meet the changing needs of organizations. In 2024, key additions include a new data catalog experience, a federated governance approach, and advanced tools for data quality and health management. These features have made Microsoft Purview a more powerful and versatile solution for Data Governance. Besides this, Microsoft has partnered with leading Master Data Management (MDM)  specialists, such as Profisee, to integrate with their platform.

Adoption and Impact

Thanks to its ability to adapt to the latest trends in data management, Microsoft Purview has been widely adopted across industries such as finance, healthcare, retail, and manufacturing. Its comprehensive set of tools and features has helped organizations improve their Data Governance practices, enhance data quality, and ensure compliance with regulatory requirements. The close integration with Microsoft products like Azure and Fabric makes it the catalog of choice for many companies.

Future Outlook

Looking ahead, Microsoft Purview is poised to continue its evolution, with ongoing enhancements and new features aimed at addressing emerging Data Governance challenges. They will seek to create more efficiencies in managing metadata, expanding data quality functionality and increasing the use of AI. As organizations increasingly rely on data to drive their business strategies, Microsoft Purview will be crucial in helping them manage and govern their data effectively.
But for now, let’s look at the new features for 2024!

2. New Data Catalog Experience

Microsoft Purview is introducing a new data catalog experience designed to enhance data governance and create business value. This new experience integrates various solutions into a single, cohesive SaaS framework, making Data Governance more accessible and efficient. The federated governance approach centralizes key strategic and administrative components while allowing for distributed responsibilities, self-service access and maintenance. Enhanced features include governance domains, data catalog access policies, critical data elements, glossary terms, data products, data quality and health management features, all aimed at improving data management and discoverability. 

2.1    Federated Governance Approach

The federated governance approach in Microsoft Purview is designed to balance centralized control with decentralized execution, providing a flexible and efficient way to manage Data Governance across an organization. This approach combines the strengths of both centralized and decentralized models, ensuring data safety and quality while allowing for self-service access and maintenance. Let’s examine the key elements that define this governance structure:

• Centralized Control: At the core of the federated governance approach is the centralization of data safety and quality standards. This means that overarching policies, standards, and guidelines are established at a central level. These standards ensure that data across the organization meets consistent quality and security benchmarks, crucial for maintaining data integrity and compliance with regulatory requirements.
• Decentralized Execution: While the standards and policies are centrally controlled, the execution of these policies is decentralized. This means that individual departments or business units have the autonomy to manage their data within the framework of the central standards. This decentralized execution allows for greater flexibility and responsiveness, as departments can tailor their Data Governance practices to meet their specific needs and objectives.
• Integration with Business Concepts: The federated governance approach also involves organizing data by business domains, such as Marketing, Finance, or Operations. This organization aligns Data Governance with business objectives, making it easier for users to find and use data relevant to their domain. By structuring around business concepts, organizations can ensure that data is more accessible and meaningful to users.
• Benefits of Federated Governance: The federated governance approach offers several benefits. It enhances data quality and security by maintaining central standards while providing the flexibility needed for departments to manage their data. It also promotes greater data accessibility and usability through self-service tools, enabling users to leverage data more effectively. Additionally, by aligning with business concepts, it ensures that Data Governance practices support organizational goals and objectives.

In summary, the federated governance approach in Microsoft Purview strikes a balance between centralized control and decentralized execution. It ensures data safety and quality through central standards while providing the flexibility and autonomy needed for effective data management at the departmental level. This approach enhances data accessibility, usability, and alignment with business objectives, making it a powerful framework for modern Data Governance.

2.2    Governance Domains

Governance domains in Microsoft Purview are designed to organize data by business concepts, making it more accessible and aligned with organizational goals. This approach helps ensure that Data Governance practices are relevant and meaningful to different parts of the organization, facilitating better data management and usage. Governance Domains facilitate effective data management by:

• Organizing Data by Business Concepts: Governance domains categorize data based on business functions such as Marketing, Finance, Human Resources, and Operations. This categorization aligns with the specific needs and objectives of each business unit. For example, the Marketing domain might focus on customer data and campaign performance metrics, while the Finance domain would prioritize financial records and compliance data.
• Improving Data Accessibility: By organizing data into governance domains, Microsoft Purview makes it easier for users to find and access the necessary data. Users can navigate through the data catalog based on business concepts, quickly locating relevant datasets without having to sift through unrelated information. This streamlined access supports more efficient data usage and decision-making.
• Enhancing Data Relevance: Governance domains ensure that Data Governance practices are tailored to the specific requirements of each business unit. This relevance is crucial for effective data management, as different departments have unique data needs and priorities. By aligning with these needs, organizations can ensure that data is managed in a way that supports their overall business objectives.
• Facilitating Collaboration: Organizing data by governance domains also promotes collaboration across different parts of the organization. When business concepts categorize data, it becomes easier for teams to share and collaborate on data projects. This collaborative approach helps break down data silos and encourages a more integrated and holistic view of the organization’s data assets.
• Supporting Compliance and Security: Governance domains help ensure that Data Governance routines comply with regulatory requirements and security standards. By categorizing data based on business functions, organizations can apply specific compliance and security measures to different types of data. For example, financial data might require stricter security controls and compliance checks compared to marketing data.

Let us illustrate that with some examples. In practice, governance domains can be customized to fit the unique structure and needs of an organization. A retail company might have governance domains for Sales, Inventory, Customer Service, and Supply Chain, each with its own set of Data Governance policies and standards. A healthcare organization might have domains for Patient Records, Billing, Clinical Research, and Regulatory Compliance.

In summary, governance domains in Microsoft Purview organize data by business concepts, improving accessibility, relevance, and collaboration. This approach ensures that Data Governance practices are aligned with organizational goals and tailored to the specific needs of different business units. By categorizing data this way, organizations can enhance data management, and support compliance and security, while facilitating more effective data usage.

2.3    Data products

Data products in Microsoft Purview are designed to group related data assets, making them easier to discover and access. This concept helps streamline data management and enhances the overall user experience. Here are the key aspects of data products:

• Grouping Data Assets: Data products allow you to bundle related data assets, such as tables, files, and reports, into a single product. This grouping simplifies data discovery and access, as users can find all relevant data in one place.
• Easier Access: By organizing data into products, users can request access to a single data product instead of multiple individual assets. This reduces the administrative burden and speeds up the process of gaining access to necessary data.
• Enhanced Discoverability: Data products improve the discoverability of data assets by providing a clear and organized structure. Users can easily navigate through the catalog to find the data products that are relevant to their needs.
• Alignment with Business Concepts: Data products can be aligned with specific business processes or domains, such as Marketing or Finance. This alignment ensures that data is organized in a way that supports business objectives and enhances its value.

2.4    Data Quality 

Data quality is a fundamental aspect of effective Data Governance, ensuring that data is accurate, complete, reliable, and relevant. High-quality data is essential for making informed business decisions and driving operational efficiency. In Microsoft Purview, a variety of tools and processes are implemented to maintain and enhance this quality.
Key Processes for Enhancing Data Quality 

The key processes that Microsoft Purview applies are: 

• Data Profiling: Getting to know your data is the first step in any data initiative and it’s even more applicable when you want to start monitoring your data for data quality. With key statistics like distribution, uniqueness, and min/max values, you can get insight into the columns of your dataset.
• Data Quality Rules: A key component of maintaining data quality is the ability to define and enforce specific data quality rules. Within Microsoft Purview, you can define and enforce data quality rules. These rules specify the criteria that data must meet to be considered high quality. For instance, a rule might require that all customer records include a valid email address and phone number. By enforcing these rules, organizations can ensure that their data remains consistent and reliable across different systems and processes.
• Continuous Monitoring: Data quality is not a one-time effort but an ongoing process. Continuous monitoring of data quality metrics helps organizations track the health of their data assets over time. Microsoft Purview provides dashboards and reports that offer real-time insights into data quality, enabling data stewards to identify trends and take proactive measures to address emerging issues.

Impact on Decision-Making

These processes directly influence the reliability of analytics and reporting. High-quality data produces more accurate, actionable insights. This, in turn, supports better decision-making and helps organizations achieve their strategic objectives. By maintaining high data quality, organizations can trust their data and use it effectively to drive business success.

Data Quality Support

Microsoft Purview currently supports data quality management for a variety of sources, including:

• Azure Data Lake Storage (ADLS Gen2)
• File Types: Delta Parquet and Parquet
•  Azure SQL Database
• Fabric data estate in OneLake (including shortcut and mirroring data estate).
• Mirroring data estate: CosmosDB, Snowflake, Azure SQL
• Shortcut data estate: AWS S3, GCS, AdlsG2, and dataverse
• Azure Synapse serverless and data warehouse
• Azure Databricks Unity Catalog
• Snowflake
• Google Big Query (Private Preview)

In essence, Microsoft Purview provides essential tools and processes to ensure consistent, high-quality data across the most common analytics platforms, with the flexibility to support additional sources as the platform continues to evolve. 

2.5    Health Management

Health management in Microsoft Purview focuses on maintaining and improving the overall health of data assets. This involves implementing health controls and actions to ensure that data remains in good condition and fit for purpose.

• Health Controls: Health controls are predefined rules and checks that monitor the health of data assets. These controls can detect issues such as missing values, duplicate records, and data anomalies. For example, a health control might flag a dataset with a high percentage of missing values, indicating a potential quality issue that needs to be addressed.
• Health Actions: When health controls identify issues, health actions are triggered to address and resolve them. These actions can include automatically cleaning up duplicate records, alerting data stewards to review and correct data anomalies, or initiating data cleansing processes. Health actions help maintain the integrity and reliability of data assets by proactively addressing issues as they arise.
• Continuous Improvement: Health management is an ongoing process that requires continuous monitoring and improvement. Microsoft Purview provides tools to track the health of data assets over time, enabling organizations to identify areas for improvement and implement corrective measures. Continuous improvement helps ensure that data remains in good condition and supports the organization’s Data Governance goals.
• Supporting Compliance and Security: Effective health management also supports compliance with regulatory requirements and security standards. By maintaining good data health, organizations can reduce the risk of data-related issues, such as errors in financial reporting or breaches of sensitive information. This, in turn, helps protect the organization from potential legal and reputational consequences.

In summary, while data quality and health management are critical components of effective Data Governance, they are just part of the broader framework that Microsoft Purview provides. Through federated governance, organizations can balance centralized control with decentralized execution, aligning data governance with business needs across governance domains. Data products help streamline data management and discovery while focusing on high data quality and proactive health management ensures that data remains reliable, secure, and fit for purpose. Together, these features empower organizations to enhance decision-making, maintain compliance, and achieve their strategic objectives, all within a flexible and scalable governance framework.

3. Pricing

Microsoft Purview does not only come with new features but also with a new pricing model. Starting November 1, 2024, Microsoft is introducing a new pricing model for Purview Data Governance. This change aims to better align costs with actual business usage and streamline billing processes. The new model replaces complex pricing schemes with a structure based on practical Data Governance activities.

Two main components drive the new pricing:

1. Data Catalog: Charges are based on governed assets, meaning you only pay for assets actively managed within Purview’s governance framework, such as those linked to data products or critical data elements. Governed assets are billed at $0.0165 per asset per day, translating to roughly $0.50 per asset per month. 
   Only assets linked to governance concepts count, making it a pay-as-you-go model. For example, if your company manages 1,000 out of 5,000 assets in a month, you'll pay $500 for that month. You won't be charged for assets not actively governed, helping keep costs in check. Each asset is also counted only once, even if it is assigned to multiple data products.
2. Data Management: Purview introduces Data Governance Processing Units (DGPU), representing 60 minutes of compute time used for tasks like data quality and health management. The pricing for DGPUs depends on the selected performance tier: Basic ($15 per DGPU), Standard ($60 per DGPU), or Advanced ($240 per DGPU). For example, if an organization runs 100 data quality rules in a day, consuming 0.02 DGPU per run with the Basic SKU, the total DGPU usage would be 2 units, costing $30.

To help summarize, here's a quick look at the costs:

Until November 1^st, 2024, the current pricing remains in effect. This includes charges based on capacity and number of scans, with no fees for Data Quality testing — available for current use at no extra cost. 

Afterward, scan and data map costs will be abolished, although charges for the Self-hosted Integration Runtime (SHIR) and private endpoints will still apply. This pricing will be implemented across multiple regions, enhancing cost transparency and enabling efficient scaling of Data Governance while maintaining expense control.

Conclusion

If you've considered Microsoft Purview in the past and felt it wasn't quite right for you, now is the perfect time to take another look. Microsoft Purview has undergone significant changes, evolving into a more powerful and flexible tool for Data Governance. The 2024 updates bring a fresh data catalog experience and a federated governance approach that could change your perspective on what Purview can do for your organization.

These enhancements make managing data more straightforward and intuitive, allowing businesses of all sizes to align their Data Governance with their strategic goals easily. Plus, the new pricing model starting November 1, 2024, ensures that you only pay for what you use, making it cost-effective to scale as your needs grow.

Microsoft Purview has experienced a renaissance, transforming into a tool that meets the demands of today’s data-driven world. Give it another look—Microsoft Purview might now be the key to unlocking the full potential of your organization's data.

Ready to see how Microsoft Purview can revolutionize your Data Governance? Contact us today to schedule a demo or to discuss how Purview can be tailored to meet your unique business needs.