Modern Overview of the MIT CDOIQ Symposium

 

Modern Announces Partnership with Data Mesh Pioneers, ThoughtWorks 

 

In July, we collaborated with ThoughtWorks at the annual CDOIQ Conference in Cambridge, MA to discuss real-world Data Products implementation and best practices for Data Mesh. The data community, especially CDOs, emphasized the importance of raising awareness and gaining clarity about data products. 

Surprisingly, the focus was more on the underlying data stack rather than Generative AI, which has been a prominent topic in other conferences this year. The data stack is the powerhouse behind AI/ML stacks and data applications.

View the session: Shaping the Future of Data: Harnessing the Power of Data Products and Data Mesh with Unified Architectures 

 

Data Hot Topics 

 

Transform Data Team from an Overshadowed Backend Engine to a Profit Hub 

In the past decade, we focused on establishing data teams and processes, but amidst innovation, the business impact was sometimes overlooked. However, the truth remains that a data team’s primary purpose is to drive profits for the organization. 

• Feedback Loops with Business: Aligning all data activities with business objectives and key performance indicators (KPIs) is crucial. In real-world scenarios, data teams often lack transparency on business goals and the impact of their efforts on end consumers and downstream teams. This calls for a shift in approach, prioritizing people and data culture within the organization while leveraging technology and process improvements. To be truly data-driven, data must take center stage in the business.
• Collaboration with the CFO’s Office: The CFO’s office is focused on revenue, profit and loss (P&L), and return on investment (ROI). To understand the impact of data on various initiatives, close collaboration with the CFO’s office is vital. Decisions and priorities should be made collaboratively, and data teams should actively contribute to consumer-facing areas such as sales, customer success, and marketing, where data significantly impacts customer touchpoints and operations. Aligning data initiatives with the CFO’s office priorities will facilitate effective prioritization and acceleration.

 

Consensus in Data Products: The first direct bridge between Data Initiatives & Business

Like all good things, administering Data Products too is a combination of technology, culture, and process. 

• People Angle: To treat data as a product, high-quality product managers must be assigned to the case. Data needs a product mindset that addresses consumer pain points and drives business-driven initiatives. Data teams can become consumed by maintenance and self-induced innovation, losing sight of ROI and consumer needs. Often, Data Producers are unaware of how their data is being consumed or impacting downstream operations. Data Products enforce the adoption of a product mindset, ensuring data is consciously produced, and initiatives are ROI-driven and consumer-focused.
• Tech Angle: Data Products encompass all the attributes we’ve always desired for data: reliability, high-quality, and optimal governance. It’s a concise branding that encapsulates the DAUNTIVS* attributes in one straightforward package. The key distinction between traditional processes and the data product approach lies in direct impact loops with business outcomes. Achieving such flexible loops requires interoperable code, declarative change management, and independent infra provisioning (via IaC), akin to Software 101. By omitting the complexity and debt of data pipelines resulting from multiple fragmented tools in the ‘MAD Ecosystem,’ these loops can be efficiently maintained.  
• Process Angle: Administering Data Products involves a combination of technology, culture, and process. Traditionally, data initiatives have been somewhat non-deterministic due to the dynamic nature of data. Data Products introduce a change by incorporating more deterministic software practices. Unlike prevalent left-to-right processes, Data Products enforce a right-to-left approach where data producers are accountable for the data they produce, and data consumers regain control over data, enabling them to enforce their requirements effectively. 

 

Data Governance

Data Governance emerged as a popular topic among attendees, even though not extensively covered in sessions. Many in-person and virtual discussions focused on establishing the governance aspect, considering the involvement of authorities and the introduction of laws, bills, and acts globally to ensure data and AI system reliability and governance. 

While Data Product sessions didn’t delve deeper into governance, including our session, it’s essential to highlight that Data Product paradigm consistently addresses data governance as a foundational pillar. Unified governance, particularly challenging in fragmented data stacks with numerous tools and data assets, is addressed through the infrastructure piece of Data Products. Containerization or isolation of goal-driven verticals enables different tools/capabilities to interact and rely on a common governance engine that standardizes policies across the data stack. 

 

AI/ML: Transition from Good-to-Have to Must-Have 

AI is undeniably here to stay. While it may not have been the star of the show, its significance cannot be ignored. In the current ecosystem, AI has transitioned from a good-to-have to a must-have competitive feature. However, its true potential lies in the quality and reliability of the data that fuels AI models and applications. This is where Data Products play a vital role, empowering organizations to scale their AI initiatives with more dependable and consistent models. 

  

How DataOS is Approaching Data Products 

 

DataOS, an integrated platform, streamlines and expedites data development cycles. Equipped with comprehensive tools, teams can easily build, manage, deploy, and iterate on data products while ensuring seamless compatibility with existing data infrastructure. This empowers businesses to maximize the value derived from their data assets without interruptions. 

Thoughtworks, with its expertise in data strategies, has led successful data transformation initiatives. Teaming up with the technologically advanced Modern Data Company’s DataOS Platform, designed to streamline Data Products creation and management, forms the backbone of this partnership, crucial for a Data Mesh implementation.  

“The partnership between Modern and ThoughtWorks marks a significant step as we transform how data is implemented and applied across an organization. We’re changing the game by moving from traditional tables to a Data Product approach, and this collaboration significantly advances our shared vision. At The Modern Data Company, we firmly believe that the future lies in viewing and treating data as a product. This reimagining enables rapid, comprehensive creation and management of data, accelerating innovation and unlocking its full potential,” said Srujan Akula, CEO, The Modern Data Company. 

“The Modern Data Company and Thoughtworks have partnered to combine the world’s first data operating system, Modern’s DataOS, and Thoughtworks’ world-class data engineering and AI practices to help you thrive in today’s data-driven economy,” said John Spens, VP Data & AI Service Line, Thoughtworks. “Accelerate insights to drive your business by delivering transparent, trustworthy and accessible data efficiently and well.” 

 

Read more about our partnership.

View the session: Shaping the Future of Data: Harnessing the Power of Data Products and Data Mesh with Unified Architectures