MLOps in Government: What You Need Beyond Code to Get Your Model in Production

“This is great!  Will it be ready by the quarterly review in two months?,” my customer asked.

“It’s going to be ready by Tuesday,” I thought.  But before saying anything, I took a pause.  I knew my customer, an SES member with over 30 years of experience, must know something I didn’t, either about the technology limitations of working within the agency’s firewall or the cultural limitations that were going to hold back presentation and adoption of this tool.

“Sure,” I said, “it’ll be ready by then.” As I walked back to my desk to start work on some other projects I had going on in parallel, I continued to think about this.  Why would this take two months to finish?  The code was done, the use case was demonstrated.  What was stopping us from presenting this to agency leadership, or at least a broader group of possible users immediately?

What do you know? While the project was well received and adopted following the quarterly review, it really did take two months to finish.  Not to finish the code (which was basically done when I showed the product to my customer) or to demonstrate the use case (which was essentially self evident).  It took two months to brief all the relevant parties, explain to the skeptics why this was needed, get people onboard who had put their capital behind other solutions, and so on and so on.  All essential tasks that we needed to complete so that when it was presented at the quarterly review it was just setting the ball on the tee and knocking it out of the park.

Think Before You Code

This piece is about the things you need to be thinking of before, you start typing and spinning up clusters; what pitfalls to avoid and best practices to adopt when deploying a machine learning models in production for federal government customers if you want to be successful. If you’re looking for help on code and architecture, there are a lot of great resources out there (here, here, and here are just a few to get you started).

Before your project even gets going you need to ask yourself if machine learning is even the right solution to your problem. A variety of nontechnical reasons can influence starting an MLOps project—maybe your customer’s agency just bought a new system she wants to take for a spin or you want to test out that latest hugging face algo on the job. But if your problem can be solved using a lower cost, rules based, programmatic approach, you would be wise to go with it. Choosing the simpler solution is almost always more successful in the long run.

Then,  if you’ve determined that MLOps is the right solution to your problem, try to pick the simplest model that can effectively address the problem. Deep learning and large language models can be incredibly powerful. But you need to determine if the increase in predictive power is worth the tradeoff of using a black box model that will be hard or impossible to explain to a client. If a simple, explainable model like linear or logistic regression will work, you should probably stick with that.

Now that we’ve addressed those concerns, let’s dig into the specific concepts you need to embrace to ensure MLOps success for federal government customers.

Focus on Governance

No surprise that when delivering to the government, you need to focus on governance. In the world of MLOps, governance is critical, especially in a government setting. As a starting point, you need to understand the rules and regulations that pertain to your contract, specific project, and agency and ensure that you comply with them. Governance also extends to areas such as version control, standardization, and traceability, which can help ensure that your project is transparent and accountable. In many cases, this is a regulatory necessity, but it also promotes trust among your stakeholders. Remember, in government, unlike in many other industries, the stakes for accountability and transparency are high, and the consequences for failure can be severe. MLOps in government don’t drive SEO to traffic to a website or make better recommendation engines; they drive decisions that directly impact the finances, health, and security of the nation.

Have an Ironclad Data Protection Plan

Data is the lifeblood of machine learning models, but it can also be a source of risk. In government, the sensitive nature of much of the data that is handled adds an extra layer of complexity. You must ensure that your data protection protocols are robust, compliant with all relevant regulations, and capable of protecting against both internal and external threats. This might involve encryption, anonymization techniques, and secure data storage and transmission practices. It's also critical to have a response plan in case of a data breach. This is not just about protecting your project, but also about safeguarding the privacy and trust of the citizens served by your agency. Read that all twice when working in a classified environment.  You don’t want to be tangentially responsible for data getting into the wrong hands.

Identify and Leverage Allies

In any organization, but especially in government, getting buy-in for a new project can be challenging. To help navigate this, identify allies within the organization who understand and support your work. These allies can be instrumental in overcoming resistance, promoting your project, and helping you navigate the political landscape. They can also provide valuable insights into the culture and dynamics of the organization, helping you better understand how to present your project in a way that aligns with these factors.

Understand Your Agency-Specific Technology Limitations

Every agency is unique, and so too are its technology limitations. What works in one agency might not work in another. You need to be well-versed in the specific technology landscape of your agency, which includes understanding its existing technology stack, infrastructure, and the resources that are available to you. You also need to be aware of any regulations or policies that might impact your project, such as rules around the use of certain types of software or restrictions on cloud-based solutions. By understanding these limitations, you can design your project in a way that aligns with your agency's capabilities and constraints.

Spend A Lot of Time Educating Skeptics

Resistance to new technologies or methodologies is a common challenge in many organizations, and federal government agencies are no exception. It's important to be patient and invest time in educating those who are skeptical about your project. This includes explaining the benefits and the rationale behind your project, addressing concerns, and demonstrating how your project aligns with the goals and values of the organization. Remember, skepticism is often rooted in misunderstanding or fear of the unknown. By addressing these underlying issues, you can help win over skeptics and turn them into advocates for your project.

May the Force be with You

Navigating the world of MLOps in government can be a daunting task, marked by unique challenges and hurdles. However, by focusing on areas like governance, data protection, leveraging allies, understanding your agency's specific technology limitations, and educating skeptics, you can significantly increase your odds of success. Remember, while the technical aspects of deploying a machine learning model are undeniably important, the softer skills associated with managing the cultural, political, and regulatory landscape are equally vital.

In government, the stakes are high and the path is often complex, but the potential rewards are vast. By integrating machine learning models into our government agencies, we have the opportunity to revolutionize the way we deliver services, make decisions, and serve the public. So arm yourself with the right knowledge, approach each project with a keen sense of strategy, and don't be afraid to meet challenges head-on. After all, in MLOps, as in government, perseverance, vision, and adaptability often pave the way to success.

Benjy Braun is Vice President of Data Solutions and Innovation at Capital Technology Group.  He has over a decade of experience in the govtech space where he delights in transforming abstract ideas into tangible, data-centric platforms that empower customers to understand, make decisions, and act.