Build trust in AI technologies with this vital framework

Lucy Huang, Product Manager at FullStory, gave this presentation at the Product-Led Festival for our sister community Product-Led Alliance.

I’m Lucy Huang, and I'm a Product Manager at FullStory, your go-to spot for understanding your digital experience.

On a personal note, my career has been focused a lot on health and safety, privacy, integrity, risk, you name it. Basically, I spend a lot of my time thinking about all the things that could go wrong, prioritizing what to work on first, and shaping the policies and procedures within organizations to manage that risk.

Here's a primer on what we're going to cover today: managing risks and AI technologies. All opinions are my own, except when I pull in some headlines to highlight what's going on in the industry. It's a very fast, changing place.

It's probably been hard not to notice all the tremendous advances in AI recently, especially machine learning and generative models.

Today, we'll talk about those advances in machine learning, and what AI governance frameworks you can apply to manage your risk, user privacy, and ethics.

So to kick it off, here are two anti-goals that I don't want you to come away with from this.

Number one, I'm not here to fearmonger, but that doesn't mean there isn’t a very real risk that we’re responsible for as shapers of product and messaging this to the market and our customers.

Secondly, I'm not a machine learning engineer or lawyer, but I’m here to talk about the risks of AI in SaaS, and that should show that even you can start to contribute to your organization's policy and procedures governing AI.

• The emergence of generative models
• Why generative AI is like riding in a car
• The risks and challenges associated with generative AI
• The AI risk management framework
• So how will these actions translate?
• Prioritize trust in product management for AI

The emergence of generative models

We'll start with a little bit of history. In 2022, we actually saw a really tremendous advance or spurt in generative models, and along with that, there was surprisingly open distribution and access.

As a high-level primer on generative models, these are different from discriminative models that were more widely used in previous aspects of data science.

Discriminative models are a class of supervised machine learning models that make predictions by estimating conditional probability. We won't get into the math of it too much, but TL;DR: they can't generate new samples, it's more of a ‘if this, then that’ logic, used for classification tasks where we might use x features to classify to a particular class y.

One example is email spam. That might be a simple yes or no label for this Email Inspector that you're building.

Now we've moved on to the era of more generative models, which are a class of algorithms that make predictions by modeling out joint distributions. There are a lot more steps involved here to take the probability of the class and the estimated distributions.

But again, the TL;DR: they take input training samples and learn a model that represents that distribution.

So again, taking that email spam example, generative models can actually be used over time to generate emails that could even fool the Email Inspector. So the twist is that over time, the generative model can gradually fool a discriminator or that email yes or no spam inspector we've talked about.

And that's what we're seeing today in more recent advancements. If you take that specific flavor of generative models, we have large language models (LLM) that use deep learning in neural networks, such as ChatGPT.

We also have text-to-image models such as DALL.E that incorporate computer vision and natural processing. We've even seen text-to-video projects come out from Meta, which takes it a little bit further than text-to-image.

There's a lot of really interesting technology out there that I’d urge you to try out.

Why generative AI is like riding in a car

Now we'll go into one of the initial risks. One of the risks I'm going to talk about is copyright.

Earlier, I mentioned that the distribution of these technologies was surprisingly open. We'll take the analogy of cars first of all because I'm assuming that everyone has driven or ridden in a car at some point in their life.

Everyone has to get a driver's license to make sure that they're qualified to drive. You have to understand the policies and procedures of the road. There are also different types of licenses to show that you have knowledge of a specific vehicle.

In addition, we have seatbelts and speed limits to protect ourselves and others from harm. There's also signage on the road, so that provides notice and transparency.

And with the democratization of generative AI, we're actually giving these cars to a wider audience than ever before. But here, the driver's test is optional.

For example, ChatGPT. How many folks have tested out open beta there? If you're familiar with Midjourney, another text-to-image service, they're actually available via a Discord server bot that has millions of users.

Personally, I'm all for the wider spread use of AI and access by different audiences, but we need to recognize that there are guidelines required. Where are the seat belts and speed limits? And who's volunteering to use them for generative AI?

There aren't a clear set of guidelines for the purposes of generative AI today, how it should be used, and how it can be measured. And honestly, this isn't much of a surprise given that the US is already one of the largest countries without significant federal data privacy laws.

To take it back a little bit, most organizations found that the onset of GDPR actually helped them build a clearer, more distinct organization, organized around managing consumer transparency and privacy.

As frustrating as it probably is for us to see all those cookie banners today, we've still raised the tide for all ships and humans on them with that set of standards.

A Deloitte survey found that 44% of consumers felt that organizations cared more about their privacy after GDPR came to fruition. And even now, Europe is leading the way with the first proposed AI Act, which is the first set of regulatory frameworks for AI governance.

So I think today we're seeing that folks are being given cars without a seatbelt and being told to drive and explore generative AI. With this great power comes great responsibility, and you and your organization should include that within your AI and product strategy.

The risks and challenges associated with generative AI

Now we'll go into the copyright piece that I touched on a little bit earlier.

Here is a headline from the New York Times:

‘An AI-Generated Picture Won an Art Prize. Artists aren't happy.’

So here, a digital artist actually entered into an art contest in Colorado, in the Digital Arts category, came first place, and won $300. They actually used Midjourney, which I talked about before. It's a service that's available on Discord that provides text-to-image renderings.

So here, the digital artist actually took these renderings from Midjourney and made significant adjustments to these images in Photoshop until he was satisfied. He enhanced the resolution of the images using a tool called Gigapixel and ended up submitting these pieces on canvas.

They're now listed for sale at $750 a piece, based on their assessment of fair market value.

With all these advances in technology, the question comes up: what makes AI different from a camera that captures the presence of someone else's creation? And here, the answer is copyright.

‘AI-created images lose US copyrights in test for new technology.’

The above headline is significant for Reuters because it documents one of the first decisions by a US court or agency on the scope of copyright protections for AI-created works.

This example is relating to the images in a graphic novel that were again used by that same AI system, Midjourney The US Copyright Office actually ruled that the images in this graphic novel should not have been granted copyright protection.

Granted, the text was made by the author, but not the images. And though the author did mastermind the prompts for this text-to-image generation of the images, the author ultimately didn’t create these images.

So how does it relate to you and your customers?

Takeaway number one is that your customers will be under scrutiny for using AI tools and services that you provide. So how can you protect your customers from that risk?

Takeaway number two is that you need to avoid the use of these protected datasets or find ways to partner on the royalties on them. Otherwise, your monetization strategies will be impacted. I suggest partnering directly with creators or the sources of this training data to ensure that your value chains are protected.

As a note, the space is constantly changing. Just last week, we had some notable names sign an open letter that called for a six-month ban on creating AI that's more powerful than GPT-4.

This open letter was for certain founders of OpenAI to solidify their lead in the market. I'm not really here to comment on that. My intentions are more to highlight the risks of AI on society as a whole, how this has been recognized by leaders, and the gap of action to address that.

So this open letter says that AI systems such as GPT-4 are now becoming human-competitive at general tasks. There are risks for such systems that could be used to generate misinformation on a massive scale, as well as the socio-economic impact of potential mass automation of jobs.

Again, it's not my intention to fearmonger, but it’s important to at least be aware of the risks such as the prevalence of deepfakes and additional tools of generative AI. There are more tools available for bad actors to use.

So your trust, safety, and integrity will also need to uplevel themselves to understand and combat the malicious use of these tools.

If you're in the space already, you're pretty familiar with how quickly bad actors and fraud rings can level up. They're really agile. So here are some examples of what to consider.

In the risk and fraud space, take identity verification. For example, submitting driver's licenses. These could also be generated by AI or certain PII pieces as well.

We also have social media. The prevalence of bots is already a huge issue. Imagine bots having access to these large language models that are able to replicate human speech, language, and communication at a higher level than before.

Extremist groups or other unsavory characters might take advantage of these tools to further their agenda on your community platforms.

And then the last example is financial services. Here, we’re thinking about account takeovers or these scammers that are tricking unsuspecting folks to reveal information with really sophisticated pre-generated chat scripts.

The AI risk management framework

So now that we've covered how the unexpected uses of AI by bad actors might induce risk, we'll move on to risk management frameworks. How can we address these?

Today we'll walk through the initial traditional risk management frameworks. And then next, we'll talk about how this evolves in AI governance.

This is following guidance from the US Department of Commerce's National Institute of Standards and Technology.

You’re probably already familiar with the essential activities to prepare your organization on how to manage security and privacy risks.

One, categorizing the system and information that's processed. Where’s it stored? How’s it transmitted? Running an impact analysis on that, selecting controls to actually protect your system based on that initial impact analysis, and implementing and documenting those controls.

Over time, you’ll also need to assess and audit if those controls are actually in place, if they're operating as intended, and producing the desired results. You’ll also need senior officials to authorize the system to operate.

And then lastly, continuously monitor that control implementation and any additional risks that might come up for your system.

For all of this, you really need a higher-level set of policies and procedures to guide your organizations where software can’t. Ideally, you have all three of those, though,

I think really common themes that you can pull out here are that data classification piece, and also establishing the provenance of that data.

One thing I want to bring up is that there aren’t really any significant regulatory frameworks regarding AI governance today. And it's more likely that the EU will get to this first, just like they did with GDPR. In fact, I think the EU has already proposed the first set of regulatory frameworks called the AI Act. I really recommend reading it.

Some recommendations around governmental action in that space are to establish new authorities or agencies that are capable of tracking and overseeing the development of advanced AI, and also the large data centers that are used to train it.

There are also potential recommendations to watermark or establish the provenance of AI-generated content.

And then lastly, liability. What happens when AI harm is caused? And additionally, support to increase public funding for AI safety research.

On a separate note from those recommendations and the AI Act, we’ll actually get into the AI risk management framework. This is version one that's been shared by the same National Institute of Standards and Technology.

Again, I’d recommend keeping abreast of this space because it changes incredibly fast. GPT-3 was being talked about just a few months ago, and now GPT-4 is available to users as well.

Again, this is just version one. The goal of this risk management framework, published by NIST, is really to cultivate trust in AI technologies, which is necessary if society as a whole is to widely accept AI.

The core of this framework describes four specific functions. In the center, you have ‘govern,’ with ‘map,’ ‘measure,’ and ‘manage’ around it. This is to help organizations address the risks of AI systems and practice. We'll talk through each of these functions and how they're applied in context-specific use cases and throughout the stages of the AI lifecycle.

Govern

At the center of this is really building those initial policies, procedures, and processes. You want to make sure that a culture of risk management is cultivated and present within your organization.

So ensure that they're able to manage, understand, and document the regulatory requirements involving AI, and be able to tie that to specific tactical policies, procedures, and steps within your organization.

And taking that a step further, tie that to the actual product experience and product design.

You also want to ensure that there are mechanisms in place to actually govern and inventory your AI systems. And as you're building your AI governance team, make sure it's a diverse team with diverse skill sets and background skills, etc.

I’d recommend maybe something you could start as soon as next week is to host tabletop exercises. Encourage your teammates to try out ChatGPT and DALL.E, and build a muscle for this type of thinking of how these types of tools might be used and governed.

Map

Next, we have the ‘map’ piece of this, which really ties things to context. How do we recognize the context of what risks matter the most?

For example, take gambling. In the entertainment industry, it's probably super okay to talk about gambling and propose products and features around it. But in other certain cases, gambling could be a more sensitive, not suitable for work type of topic.

So that context really, really matters, and by understanding that context, you can develop that intended purpose, benefits, and norms in which AI can be deployed and documented.

Again, in this map function, you also want to define the specific methods used to implement the tasks that the AI system would actually support. And here, for example, you’d want to at least outline and say, “Hey, this is using a classifier versus a generative model versus more of a recommender.” Being able to define those specific methods is really important.

As a part of the map function, you'll also want to develop internal risk controls for components of the AI system and keep abreast of any third-party AI technologies that might be used as well.

Lastly, for this map function, you also want to address the privacy and the provenance of the data used in this creation.

Manage

For the ‘manage’ portion of this framework, this is where you can provide your PM mindset, assess what risk exists, what to prioritize, and how to act based on that projected impact.

Measure

Lastly, we have the measure portion of this, which is really talking about the ways that we can enumerate those approaches or metrics of the risks of adopting AI.

You'll want to regularly assess the appropriateness and impacts on affected user groups or communities. I’d recommend pulling in domain experts and users to be consulted for their feedback. So hopefully, this all resonates with you. It's not something that's too dissimilar. But here, it's really important to outline those functions.

So how will these actions translate?

One, we want to provide respect for the original creators and artists, given the lack of copyright protections for AI-generated works today. You want to make sure to partner with those original creators to actually protect your value chain and your monetization strategies.

Secondly, protection of privacy and ethics. As I mentioned previously, you want to carefully select the initial data used to train these models to avoid including toxic or biased content.

Make sure it's originating from a source that’s given their consent towards data being used in this way, and being provided proper notice and the ability to pull out if needed.

You also want to be careful about how AI tools might be used by bad actors, for example, threats against democracy on community platforms and use in financial services by scammers.

So again, invest appropriately in your trust and safety teams. And you can start today by even just encouraging these teams to try prompt engineering themselves. It's really important to develop a familiarity with this technology.

It all comes down to reducing risks for our customers and society as a whole. Some strategies here are rather than employing an off-the-shelf generative AI model, you could consider building out smaller, more specialized models that are tuned to the needs of your organization.

I’d also recommend keeping a human in the loop to make sure that they're checking the output of generative AI before it's actually published or used.

And then lastly, really make sure that you're able to responsibly use AI and also build familiarity with it. I'd recommend avoiding using generative AI models for critical decisions such as those involving significant resources or human welfare.

While there's a need for us to be competitive and familiar with innovation in this space, we also have a responsibility to think about what impact this might have on certain communities and society.

Prioritize trust in product management for AI

So here are the takeaways. You can start all of these today. Timing is important, as we all know.

You want to prioritize trust because product management for AI is ultimately probabilistic and not deterministic.

Trust is easy to lose and hard to gain, so it's important to prioritize consumer trust, transparency, and ethical principles when building these out.

The reason this is important is because machine learning adds even more uncertainty. And the trade-off is that because of the scale that we're able to attain on machine learning, there's also going to be a small percentage of predictions that are going to be incorrect, and it's going to be really hard to understand why they're incorrect. It lacks explainability.

This is because the ML code that has seemingly similar data sets of input-output can give you wildly different results as an output sometimes.

This has really serious implications overall for the product development lifecycle, and also for software development as well, such as versioning and testing, because the data’s never really as stable as we think.

So as your product inevitably involves models that you've built, they’ll also start to drift and need to be monitored and managed. Again, tying it back to that risk management framework.

Lastly, there also needs to be that foundational governance framework for your organization's teams to be able to take into practice.

At a high level, there needs to be a partnership between community, legal, and policy teams to build this governance framework and review at least quarterly, then map that to the risk management framework that we discussed earlier in terms of map, manage, and measure.

Assign your stakeholders. Who's responsible for inputting these controls, monitoring that they're actually being implemented, and producing the desired results?

So I'm here basically today to chat about how we can all make an impact regardless of if you're a domain expert in ML or not. I think with the advent of ChatGPT-4, we've really seen how it can become accessible to more folks. And because of that, everyone has a responsibility to weigh in and carry out this AI governance framework.