The Value of Maintaining Model Activity Logs

From Development to Archive

Models and model versions are constantly churning over. Models go from development into the queue to production, and then from production into archive. At many organizations with operationalized online learning, this lifecycle repeats on a daily or weekly basis for their models.

At the same time, many organizations are not systematic in how they maintain the history for their models. Different team members keep notes on an ad hoc basis across different applications, or rely on an imperfect and unhelpful record in Git or their own written notes. 

As a result, companies can find it difficult or impossible to understand how a model changed over time, who approved changes, which version created issues, and why.  This can slow down development and iteration cycles, and also make it challenging to comply with AI regulations, demonstrate adherence to Responsible AI principles and ensure consistent model performance over time.

Maintaining an activity log for models is important for several reasons:

• Reproducibility: Tracking changes to a model makes it easier to reproduce the model later, understand why performance has changed, and debug the model.
• Compliance: Organizations need to demonstrate that a model was built and tested in compliance with regulatory requirements and Responsible AI principles.
• Collaboration: With a clear record of changes made to the model and the reasoning behind them, diverse stakeholders can better stay on the same page.
• Accountability: Understanding who changed a model and why supports the Responsible AI principle of Accountability.
• Maintenance: Data scientists can identify areas of the model that may need updating or improvement, making it easier to maintain the model over time.

Model Activity Logs in Verta Model Catalog

Verta Model Catalog tracks each model or model version as it moves through the lifecycle, recording all the changes at each stage in a Model Activity Log. For example, when a developer makes a request through Model Catalog for permission to move a model into development, the platform records who filed the change request, when it was made, and who approved the request and when.

The Verta platform also provides configurable workflow checklists to guide users through tasks related to the release of a model or model version. This could include steps for model handoffs, Responsible AI, Model Risk Management, and so on. Here, too, Model Catalog tracks each action in a checklist. For example, the platform could document that a data scientist completed model documentation and defined performance requirements for a model.

The system records all these actions and changes and more in a set of audit logs that allow authorized users to view the complete activity history for each model version – all the changes and who made the changes – in a .csv file. Any user with access to a given model can pull the log for that specific model or version, while system admins can access platform-wide logs.

Use cases for pulling the activity log could be:

• A data scientist sees that something changed in their model and they simply want to be able to understand who changed it and why.
• Governance needs to demonstrate to an auditor that the organization has a documented model release review process.
• A model produces a bad outcome – wrongly denies credit to an individual, for instance – and the organization needs to understand how the model was tested and reviewed, and who approved it and when, supporting the Responsible AI principle of Accountability.

With the growing importance of Responsible AI, regulatory compliance, and the need for transparency and accountability in AI development, maintaining an activity log is becoming an essential part of model lifecycle management. Verta Model Catalog provides an effective way for organizations to maintain comprehensive model activity logs, ensuring that they have a documented history of their models' development and enabling them to make informed decisions throughout the model lifecycle.

Learn more about Model Activity Logs in Verta Model Catalog with this 70-second video introduction.