5 Signs You Need a Model Catalog

Companies scaling up machine learning beyond a handful of bespoke models typically reach a tipping point where they need to centralize management of ML assets across their enterprise with a model catalog. But where is that tipping point? Here are five signs that your organization needs a model catalog.

#1. You have multiple model training systems used by multiple groups

Large enterprises often have multiple groups, teams or business units using different model training systems, either due to legacy decisions, the pre-existing tech stack, or because different teams use different programming languages.

It’s not uncommon for enterprises to have some models trained on SageMaker, some on Dataiku, and some on Jupiter notebooks and R studio. They also might have various third-party models in use.

Often, models across these disparate systems are not tracked at all, or they are tracked manually in spreadsheets, making it impossible to protect, reuse or govern the assets. As a result, data scientists have to “reinvent the wheel” because they have no way to discover work done in the past or elsewhere in the organization.

At the organizational level, the absence of a centralized inventory of models across disparate systems and departments makes it challenging, if not impossible, for companies to establish consistent policies and processes for deploying, managing, monitoring and governing their models, increasing reputational, legal and compliance risks.

A model catalog can address these issues by ingesting models from a variety of sources, including disparate AI development tools, and recording rich and configurable metadata, such as assigned purpose, ownership, and risk tiers.

#2. Your company has AI applications that will be impacted by the EU AI Act, Algorithmic Accountability Act or similar regulation

Government bodies around the world are actively drafting and passing bills to regulate the use of AI. These laws will likely apply to your use of AI regardless of whether your company has traditionally been highly regulated (like Finance or Insurance) or not.

Across the many new regulations, a foundational requirement is that organizations be able to generate an up-to-date inventory of all uses of AI across the enterprise, including the source of each model, documentation, and their prescribed application. With a comprehensive inventory completed, organizations also must be able to meet regulatory monitoring and reporting requirements around bias, privacy and explainability.

As a concrete example of the kinds of requirements being considered, the EU AI Act expected to be adopted this year provides that “high-risk AI systems” would require rigorous testing and risk assessment, proper documentation of data quality, logging of their activities, and an accountability framework that makes data available to authorities to scrutinize.

High-risk AI systems could include those used in autonomous vehicles, medical devices and critical infrastructure machinery, but also credit-scoring applications or recruiting tools, where there is potential for bias and risk of discrimination.

Note that the AI Act would impact organizations based outside the European Union but that do business in the EU or interact with EU citizens, similar to the GPDR data protection law. Similar requirements appear in laws under consideration in the US, such as the American Data Privacy and Protection Act (ADPPA) or the Algorithm Accountability Act, and elsewhere.

A model catalog can help an organization meet these requirements, starting with automatically documenting models according to applicable regulations, as well as providing process and audit documentation using the information stored in the catalog.

A catalog also supports AI governance through process gates for checks that must be completed before model deployment, performance monitoring for models in production with alerts when model degradation or drift is detected, and capabilities to remove models from use if they fail. Meanwhile, the data and documentation centralized in a model catalog support compliance with transparency and explainability requirements, too.

#3. Cross-team handoffs slow down your time to market and your team dreads them

Data science and ML is a uniquely interdisciplinary endeavor, with baton handoffs from data science to engineering to governance and back and forth. Since each of these functional areas has very different priorities and skills, communications at each boundary can be painful, with days spent on “simple” communications such as:

• “How do I recreate this model?”
• “What has changed between the new and old version of the model?”
• “You are missing 2 of the 3 critical responsible AI checks required for this model.”

These cross-team handoffs can easily introduce inefficiencies and delays of weeks and months, and in the meantime key contributors can move on to other initiatives and lose crucial context, introducing further delays. A central system of record for models and associated documentation can provide a “single source of truth” that considerably reduces these handoff delays.

In addition, a catalog with role-based access controls (RBAC) allows different functions to interact with model assets in a way that is appropriate to their needs, providing access to the “right” information (documentation, performance reporting, audit materials) at the right time in the right format.

#4. Onboarding for new data scientists on your team is a lengthy, time-consuming process

Onboarding new data scientists is a slow and expensive process; for a role with an average tenure of just 18 months, a 3-month ramp-up time means an 18% loss productivity out of the gate. Any reduction in onboarding and ramp-up converts directly into increases in data science productivity and more value from each hire and the team as a whole.

A model catalog with detailed documentation about each model's development (e.g., data used, feature transformation code, model training code and parameters) and lineage can significantly reduce the time needed for a new individual to onboard. This can help organizations quickly integrate new team members and avoid disruptions caused by staff turnover.

#5. ROI and efficiency of data science initiatives are increasingly under scrutiny

As the economy tightens, budgets for all functional areas are under scrutiny. Data science has been viewed for some time as a function with lots of promise, but it also has had a history of underdelivering: one researcher estimated that almost 90% of ML models never make it into production.

As a result, many executives are asking reasonable questions about the efficiency of the data science process and where more efficiency can be wrung. For instance, given the large compute (and human) cost of building a deep learning model from scratch, many large organizations mandate that before a new project is kicked off, avenues to reuse existing models be thoroughly evaluated. A model catalog can make ML assets searchable and discoverable so that existing models can be reused, whether they are in production or retired, or even if they never made it into production at all.

Similarly, management is asking for visibility into the business impact of these AI-enabled applications and efficiencies in bringing these innovations to market. A model catalog presents the natural place for such metrics to be computed, tracked and visualized.

Conclusion

To summarize, with the rising significance of AI and ML for business success, the potential risks and inefficiencies associated with their use have also increased. We have indeed reached the tipping point where organizations must take steps to ensure continuity in data science, optimize out inefficiencies where possible, and incorporate governance tools to oversee AI initiatives to mitigate risks.

To meet these demands, enterprises increasingly are adopting model catalogs as an integral part of their journey toward Operational Excellence in AI/ML, not only to ensure the success of existing projects but also to justify future investments.