AI-powered products and product-powered AI – Guy Samuel – YConf 2021

Well be talking about AI-powered products and what product-powered AI A bit about myself, so as you saw earlier in the intro, hi, my name is Guy Im from ThoughtWorks Spain, based in Barcelona Its also around lunch here, so I hope theres nothing in between my teeth or anything Ill just stand a bit further from the camera And Id like to start with what brought me into this investigation So most of us, I imagine, will have.

heard about more and more AI projects and different types of products in the fields we work in We know theres developing technology We know its being made more accessible We know its more and more common But AI, really an AI in our different products, is not such a new buzzword anymore Its something that, I believe, will be expected for us to work with in the very near future, if were not working with it already So as we jump in, I would like to ask you with this first poll, how much experience or what levels.

of experience we have with AI in our different projects? And well get a feel for a bit what kind of experiences we have in our audience today OK, excellent All right Thanks for those who have participated As you can see, so it looks like many of us havent worked with AI just yet But as we said, I think well probably be.

expected to do it quite soon So theres quite a lot of growing investment and fastly growing investment in AI You can see in this graph the investment in AI products by different companies in 2021 compared to 2020 2021, here is the dark blue And you can see a significant jump in companies dedicating more and more budget to AI, whether its research development, different types of products and projects So theres huge investment and huge growth in both the public and private sectors An interesting thing in this survey.

as well is that all company sizes are represented Its not just large enterprises, also small medium businesses Ill working more and more with AI And of course, it can be applied and were seeing it applied in just about every business sector you can imagine, whether its travel, logistics, and dynamic pricing, and data management, and customer support Just about everywhere And all of this makes me think as a business analyst product manager or product thinker, in general, what do I need to do? What do I need to know for when AI starts playing a part.

in my project? What is different about products that have AI involved? How can I or how should I adapt my product process? And all these questions, they led me to different parts of investigation So in todays talk, well also have full stories to get a feel of what its really lik with some examples from the ground But first of all, I mean, theres one thing we need to clarify before And that is like, what are we talking.

about when we talk about AI? So first of all, Ill stick to this kind of definition I wont go into all the different debates of different levels and different definitions of AI Lets focus on, lets say, the characteristics that are relevant for our work as product thinkers So one baseline definition of AI or most AI products that we see today is a system that performs a task without being explicitly programmed You dont give your model specific instructions.

on how to execute a task, such as estimating the price of a house But it rather learns that information and infers different rules and relationships from data that you feed it So one example we can see here This is a list of houses We can see the location, their square footage, how many rooms they have, et cetera, and their prices So this is used to train And this will be referred to as the training data You give it to an AI system.

And you train it to look at all the different data points in relation to the price And it will figure out how these data points, how these factors connect to create the price and will create a model to estimate the prices of future houses that maybe you dont know the price of And so thats kind of the basic two components or two sides of it So you input data In most AI projects or AI implementations that were seeing today, this is kind of the model.

thats being followed You give it data with, lets say, labels Hey, this is kind of the predictions you want to train it on And you get predictions In this case, you can feed it in your house and get an estimated price for it So those are, lets say, the two main sides of AI– the characters that are interesting for us as product thinkers.

Now, taking the same concept, you can feed rather than a list of or a data table of houses, you can feed your AI system thousands of images made up of all these pixels and trained to recognize faces You can feed it information about text in different languages and train it to perform machine translation You can feed it different elements of data tracking different systems and train it to recognize anomalies This is used of security, fraud detection, et cetera And what they have in common at these components Now, what are these components mean for us when.

were working with products? Whats different about our work versus lets say developing some enterprise software, a web application, a mobile app? So the first thing is working with data And Im sure just about everyone has heard this, right? The almost the prerequisite of one of the most important conditions or inputs for a valuable AI system with good predictions is data And this is indeed one of the biggest challenges.

were seeing across all the different projects Its not about getting just the quantity of data because you need usually depending on the application a lot of data to train the system to get good predictions, but its also about the structure the completeness and the right format of the data But as we saw its only one side of the equation You input the data And then there are many technical decisions to be made on which types of algorithms.

you use, which types of models you use, how you set the parameters And these are all AI specific decisions You wont find them in mobile apps or web apps, et cetera Now, how can we tackle these? How can we integrate these into our process as product thinkers to make great products that are powered by AI? So first of all, I think one challenge well talk about is, how to make it a product? Well, you might have these different experiments going on,.

different types of tools, et cetera How can we connect that into a flow the delivers value? Like in the talks we saw this morning Second, well dive into that question of these different decisions we want to make about the models parameters and all sorts of technical implementation, things that could affect how our product would work and how we can iterate on it Other than that, we also have decisions which are not strictly technical In this case, we might get predictions that we need to define how good they should be to be useful for me.

So there are more business decisions that would be specific to working with AI And at the end, of course, well also talk about this data challenge OK So yeah, so these are kind of the lots of questions that came up Theres obviously many more And Ill also be looking forward to your questions on other topics and how we keep learning on this But then my question to myself is, OK, what do I do with this?.

I know is coming Im working on some AI experiments And Im expecting more to come in the projects Im working on How do I tackle these challenges? Well, my proposed answers so far is to talk to people that have been working on different types of projects So well see and well get a feel hopefully for what it really looks like, how do we make these decisions, and what can we learn from experience of people.

that are already doing this So that when our time comes to dive in to AI-powered products, we will have learnt these lessons and be able to deliver value in a more effective way All right So thats what brought us here Lets talk about these different areas or these different experiences with AI-powered products And Im going to start with something that sounds– it sounds almost silly, I think, especially for this audience But AI-powered products are products They have to serve a specific user or customer need And I guess, it sounds obvious, but of course,.

as with every new technology we see, lots of hype around it People coming to us and saying, oh, I want something with AI, without necessarily thinking if thats the best solution for the current need we have So we can have different experiments, different models, and test, and make predictions about delivery routes, house prices, and things But then what do you do with them? Does it really connect to a specific customer flow? Does it answer a need that we validated?.

How does it fit into the existing customer journey? So lets take a good look at an example, I think, of this work done well Were going to talk about the work of Shimoku So Shimoku is a Barcelona-based startup They actually share an office in the building with ThoughtWorks Barcelona for those of you who still remember what an office is.

And Shimoku has several AI-based products with different value propositions Most of the work is targeted right now small medium enterprises, e-commerce, online shops And they can use different prediction models to help, a, predict customer churn, hey, which customers are you likely to lose? Another one is to qualify leads or which customers are likely to convert or to buy more And you can even predict if your demand planning matches.

your stock planning Are you going to have enough stock to meet the demand? Now, each one of these applications is actually quite sophisticated in terms of the AI behind it the several layers several different processes to make it work In my head, I kind of imagine this matrix-like screen with all the different numbers and all the data flying around And if you look into the code, its kind of like that But thats not what the customer sees.

The customer ultimately sees in this dashboard, phone numbers With a very specific clear information, you are at risk of losing these customers with this value And so this is not about showing like all the sophistication of AI Its specifically about showing what the customer is expecting to see and what they can do with it So I think this is an excellent example So your AI, it creates this prediction Its a pretty sophisticated pretty cool implementation But ultimately, the goal is to inform the decision and meet that customer need.

Now, in the case of Shimoku, they also dont stop there So we saw earlier as well in the talk about data mesh a version of this cycle– how you take data, how you generate information, extract insights from it But then the next step is action So just having the prediction is not necessarily valuable in itself And we can see that also here So we can predict not only our customers going to churn, but the system will propose what action you can take.

And actually, integrate it right there in the interface so it can be integrated with your CRM, your mailing system The action is right there for you So offering the value Im giving you, closing that loop Actually, they dont– they go even one step further So after the action, we saw the cycle that feeds back into the data And you can actually, in a separate dashboard, see the actual results of the action and get a clear measurement of the value were given.

This is the opportunities and the money that you saved or that you gained because of decisions that were supported by this AI So its clearly connecting to their customer flow, and even giving them a very tangible measure of what value they got from using this AI-powered product And so I think this was a nice example of bringing AI into the customers world Now, I think for us, this is a very good point to start.

But to be able to deliver that, as we saw, there are different types of technical questions There are many different types of models and algorithms, and we can spend a lot of time adjusting that and figuring out how to train or retrain the system Well get different types of predictions from it, et cetera Now, all this can take time And we dont always have time, or an endless amount of time,.

to deliver something Especially in critical cases like– hey, I think Im actually the first person in the conference this week to mention COVID-19, which is a bit late But yeah, its, I think, an excellent example in this case of a very urgent need So in this case, Id like to take you also to Barcelona In this case, were talking about Hospital Del Mar hospital here And I have a friend working there And they were working in the hospital in March 2020, so right at the height of the first wave of the pandemic The emergency rooms were saturated.

Doctors were overworked, sleep-deprived, and there was very, very little information about COVID-19, and very– well, one of the key challenges was being able to diagnose COVID-19 Now, in this case, the team saw an opportunity to perhaps accelerate this process At the time, the PCR tests, they took 72 hours, sometimes even a few days to give a result And of course, that wasnt fast enough to be able to control the pandemic And they seized an opportunity with readily available data they had about chest X-rays.

So they hypothesized that looking at chest X-rays, now we could use AI to generate an immediate estimation of how probable this patient is infected with COVID-19 Im going to skip right to the end of this story They were able to develop this system They were able to integrate it in the production system of the hospital You can see it here in the patient management software And this whole process of hypothesis development, testing, and integration took four weeks Right? So four weeks of overworked, sleep-deprived doctors and software engineers were able to deliver this, and deliver.

that very clear value quickly, and also keep evolving quite rapidly over time So this, in itself, I think was an impressive achievement And then I asked them, how did you do this? And they– one of the key insights, I think, that we can learn from in our own projects, maybe even if theyre not questions of life and death here, is working on the basis of existing research, best practices, and examples that are similar to your domain So in this case– and I dont think there was– we didnt.

find any other projects trying to diagnose COVID through X-rays But there are similar implementations that have been studied and refined quite well So the question is, what is this task similar to? Is it a matter of looking at an image and identifying objects? If its object recognition and location, tons of work has been done around, for example, self-driving cars So the algorithms have been there The models that work well are more or less known Is it a question of image segmentation?.

Understanding which sections of the image are there, what each element belongs to? Again, this is also something thats been studied in different domains So what they were able to do is, based on existing products and existing research, take a baseline, lets say, of the currently-known best practice of working with these types of image processing tasks, and build on that with some minimal fine-tuning to meet the requirements they had.

OK? In terms of adjusting it to the images and giving reliable enough predictions for this case And so that was also something I very much appreciate in their work Stand, really, on the shoulders of giants Here, we know, as I said, AI really is not that new already Theres a lot thats known about different tasks And if you take and seek out those examples which.

are similar to your work, you can already start with quite a high baseline Right? So that is the second story in our journey And Im going to dive into these types of questions just a bit more, of how do we actually use product thinking to help guide those decisions around how we build the AI system So one category of questions we have is, is this question even feasible? Ive tried to elegantly avoid the question of when we should use AIs in different projects So thats probably a topic for another talk But even if you hypothesize it could be good,.

theres no guarantee You know? No ones done COVID diagnosis from chest X-rays Will it work? Will it not work? I dont know Then, of course, lets say you get it working But the output here is not necessarily easy to predict or necessarily deterministic in, lets say, a predictable way So maybe your predictions will be 80% accurate, or 90% accurate, or 99% accurate How do you decide what is good enough and what is useful for your specific application? And then theres the question, how much time.

do you spend actually adjusting all these things? Training, or retraining, or rebuilding the data, versus being able to develop quickly and have fast iteration and learning cycles from them? So these are all questions when you get into the nitty-gritty of working with AI-powered products So for some of these questions, we talked to another colleague that has worked with a project here that we are bringing to you from the United States this time.

So this was several years ago And the challenge or the situation here deals with what are called Advanced Placement exams So these are taken by millions of high school students every year And these exams are then graded, and they can have quite a bit of an impact on, lets say, their college acceptance or their future path in life Whats the challenge?.

Well, every year, theres about 5 million of these exams that are taken And then you need people to properly grade them consistently, and being able to process that data in a timely way So first question, can we even do that? Can we train an AI system to grade exams similarly to a human? All right? And these are not multiple-choice exams that you can just quickly automate We want long-form answers that can be.

several paragraphs at a time So you need to be able to extract the meaning and work with that Its also quite a sophisticated use of AI technology So what do we do, as product thinkers, when were faced with this question in a novel challenge? So in this case, our answers are, lets do a proof of concept Lets do a small test And we can see with, lets say, a very basic test what kind of results we get And then with some help from our data scientists and the experts, be able to extrapolate and make assumptions.

about how we can actually improve the output we get, and decide if its good enough to keep investing in, and assume that it will solve our problem All right? Next, theres this trade-off we mentioned This discussion on how much time do we spend improving the training, the data, adjusting the different parameters, et cetera, to improve the output and the quality of our predictions And in this case, I think one of the most interesting conclusions, or most interesting tips that I discovered in exploring these projects, is a brilliant flash of product thinking.

So if we build on the concept we saw before of having one-on-ones with your key stakeholders and everyone thats involved in the process, this team, they made a new persona based not on one of the end users of the system, but on people involved in the training process of the AI So part of the value chain of delivering AI is exactly people that deal with this question How much do we spend and how do we refine the training and the adjustment of the parameters, versus the business value were expected to deliver?.

What does it mean when you apply this product approach to the people that are contributing to the AI, not just consuming it? So I have to make a decision Right? I can possibly retrain my model for a few hours, or sometimes even a few days, to improve the quality of my predictions I can also adjust different parameters to get different improvements Now, if Im– this is very much a business decision Right? How much do I spend on training and adjusting versus the speed I need for publishing, deploying, iterating, and learning from my product?.

But in most projects, its not actually something a business person can do Because then you need your software engineers, your data engineers to dig into the code, adjust the parameters, restart the whole thing, et cetera So when you want to facilitate this decision-making for them, in this case, they actually made a tool, an interface, where these personas, these actual administrators-slash-trainers, could make these decisions themselves and apply them without needing intervention in the code And this was a brilliant solution.

to help surface those kinds of decisions and help everyone take an active role in figuring out this path of, how do we ensure the best value out of our AI system Now, this case also had another upshot after deployment And so basically, the same basis of tools could then be used to monitor the system, to make adjustments based on the actual data that were seeing in the wild, et cetera, which is also, as many of you know, some of the other challenges associated.

with AI when it deals with data that maybe you werent expecting How do you respond to that, et cetera So this was treating this as a persona, as an excellent application of product thinking to meet these challenges in the world of AI One last thing that we saw in this project was also extending another thing we do as product thinkers.

Once we start a project, we want to define our measures of success on the output How will we know this product or this system is actually delivering the benefit that were expecting from it? And in this project, they extended that, and also defined measures of quality on the input And so its not just asking, whatever data I have, lets see what I can get out of it Working through this pipeline, from the end back to the start, you can define, what is the quality of data I need?.

What is the structure? What is the completeness? What is the type of variation I need to see in there so that I can be confident that Im training a valid system? OK? And this also connects a bit to the concept of data mesh that we were discussing before So quite a lot of learnings when we really, really dive into the day-to-day details of working with an AI system But applying our product thinking here, thinking of personas, thinking of our measures of quality and measures of success, doing our proof of concept, this can actually give us quite a big boost.

in working through that So that was an excellent series of investigations [INAUDIBLE] here And of course, they havent forgotten the big question of data And data– well, I think my colleagues here in the previous talks have actually made this a bit easier for us So continuing from this last thing we mentioned, when we define this measure– these measures of quality, this criteria for what I need in terms of data to train and deploy an effective AI system, Im basically defining requirements Right? What do I need my data to look like? And the next step, of course, is to think.

of that data as a product OK? Well, it looks like this is not loading Ill try this again OK OK, Ill try reloading And no, it wont In any case, well, its not a problem This kind of thinking is, of course, something that facilitates, first of all, our work with data, and it addresses one of the biggest challenges So by some estimates, some people will tell– have told me that about 80% of the effort spent on developing AI-powered products.

is spent on the data Structuring and cleaning it, defining what it needs to be When new data comes in, how do you adapt that to the pipeline? How do you retrain the model? Et cetera So this is– once you think of data as a product and you have the principles that we apply to every digital product working with your data, 80% of the problem goes away Right? Because now youll have a pipeline that will take your incoming data, structure it,.

process it the way you need it, deploy that data into your system I am now a consumer of the data I tell, through these requirements, exactly what I need And I forget about all the implementation behind it OK? So Ill have a data product feeding me exactly what I need That data product has SLAs As we discussed before, it has observability So wed know if it wasnt working well Is it not? Can I do something about it? I can easily explore and discover this data And all this is exactly discovered in the data mesh.

So the last story here– and Ill just try loading this slide up once again OK Well, its not there But one of the other examples that weve worked with quite recently is with [INAUDIBLE] So this was mentioned in the talk earlier by Kristin, [? Isca, ?] and Fatima So it worked with them There are, of course, many– as a real estate marketplace, theres.

a huge amount of data coming around and a lot of metrics So one of our projects worked– was actually concerned with building one of these dashboards to help inform decisions and actions And, as you can imagine, there are many different data sources, many transformations and updates you have to do on the data itself And initially, the project started off.

as a series of experiments, proof of concept But when it scaled, suddenly, if you wanted to add a metric, or adjust the formula for a metric, or, lets say, change how something is calculated or displayed, suddenly you had to go into hundreds, if not thousands of lines of code, figure out where it was, and hope that it didnt break anything else when you changed something And basically, transforming this into a product, the way we work with a regular application, we were able to marginalize the code into small pieces.

relevant for each domain We were able to make it testable, easy to iterate on, and easy to scale And so that is not necessarily an AI example, but the same principle applies for AI And if we go back to our very first story with Shimoku if you remember, they work with e-commerce, with small and medium businesses Small and medium businesses, how do.

they work with AI if they dont necessarily even have too many technical experiences, people to work on? Especially data engineering, et cetera? Their approach is to bring exactly that They will help you build a data product with the existing data you have, automate, and simplify, abstract away all of the things that you need, and ultimately, give you something that feeds into one of their pre-built,.

or easily adjustable AI models And that is, I think, one very elegant solution to one of the biggest challenges were seeing in all different types of AI products So if we, in the near future, are asked to work with AI or Drive projects with AI at the core or AI as a contributing factor, my conclusion so far from this is– one of the important things to keep in mind in all the challenges we face is to use our product thinking.

ahead Right? So AI-powered products are still software products So focusing on that customer experience with the personas, being able to iterate quickly and learn, make decisions This, applied to AI, tackles these specific challenges of AI So for that challenge of, how do we figure out the details on the models, you build on existing examples How do we work with the data and make sure it meets our needs?.

You define your measures of success, but also, upfront define your measures of quality on the input youre getting Ill remind you again This may be one of my favorites, thinking of personas that are not just the end users replication Everyone that is needed to contribute to that value chain of delivering AI And treating the data itself as a product And this, I think, was presented beautifully in the data mesh talk a little earlier today So these are a few tastes of what real work with AI looks like Im very excited also to keep diving into this.

and learning from the experiences and questions of the people in this group, and the things well be exploring very much in the future So I wanted to thank especially the people that contributed to the talk and shared all these stories with us so that we could learn and make better products And Im looking forward to connecting and learning more So thats the talk for today Thank you, Guy That was a great introduction to AI-led products, or product-led AI Thanks a lot Are there any questions from people in there? Please feel free to pass them.

Also, Gabe and Esra are still answering all the questions we couldnt answer before in the chat Theyre just back from lunch, so now they keep up answering all those different questions For me, a question– you started in terms with– on this product thinking slide, you started to mention about feasibility and team elements Just, if you could, elaborate a little bit more.

in terms of feasibility How would you assess that? Because I think its a black box you might open, and you dont know how fast you are, or how many resources you need How do you approach that, and what would help you to make those assessments upfront? I think technology, in general, is hard to assess upfront But it seems like it just becomes even more complex So some things are not hard to assess.

So if we look at our more well-known software products– if you show me, for example, wire frames of a mobile application or a web app, wed probably have a good enough basis to be able to estimate, is it feasible, how long will it take, et cetera And many elements of AI-powered products are similar, because all the customer flow feeding into the AI and the predictions that come out.

are still there Theyre still the same The specific challenge with AI is that I cant just assess feasibility based on a wire frame Right? And if someone comes to me and says, I have these photos, I have these chest X-rays, how good is it going to be in predicting COVID-19? How long is it going to take to get a good prediction?.

How much data will I need? And a priori, we have almost no way of answering those questions unless you try it out And you can try– in this case, for example, in the case of COVID-19, they started with just over 5,000 images of chest X-rays Is that enough to train the system? I dont know So the first step is what we mentioned before, is proof of concept The thing thats been– that weve seen thats able to accelerate and give us answers to that question of feasibility is just comparing it.

to similar projects So thats one answer we saw before And that was also applied in the COVID case Another thing which has become more and more accessible, especially in the past two or three years, is actually using pre-trained models So you dont need to be Google or Facebook with billions or trillions of photos training these super-sophisticated models Many pre-trained models that already have, lets say, some basis of image processing, natural language processing, et cetera, those are things you can take as, lets say, baselines for your task.

And then that accelerates not only the development process, but you can actually take them and build a little proof of concept, which is based on a whole lot more data than you would normally have And then you can see will it work or not So its once more like what we heard already in other talks, like, look, how you get fast to have the value.

and really see whats the value you can have in there Whats the customer value? And then also the budget you have And the faster you are to deliver that value, the more likely you see if its possible, and you see, yeah, what you get back to it So theres not much upfront planning But its really like, try to get to it very fast.

Yeah ALEXANDER STEINHART: OK, great Theres another question in the chat As its a new field and not everybody has experience with AI in the team, how do you build a team and the expertise around that? So one question, actually, I ask everyone that I was talking to in researching this talk and the different projects I work on, or at least collaborate with, is very similar to this.

Right? So in this talk, for example, I learned that we can apply our product thinking to AI questions, and AI practitioners can apply product thinking to accelerate their processes But then if we– not everyone has this experience and youre building a team, should you choose strong product people and help them learn or adapt to the world of AI? Or should you work with strong AI, machine learning, data engineering people, or data scientists,.

and help them learn the specific product thinking tips that could help them make a valuable product? I got, I think, 700 different answers Mostly contradictory Lets say my summary of what Ive heard so far is that its not a question of how– which team members you have or which capabilities you have So theres no– you cant really compromise on product thinking and the relevant level of data and AI expertise So some AI systems, some AI projects are not too complicated Right? You can actually, especially with the tools that.

make it easy, the pre-built models, existing examples, you can build something out of the box with limited technical knowledge But that product knowledge is always relevant The moment you want to adjust and optimize and scale your AI, then those technical skills are crucial So every team will have a different mix of software engineers and engineers specialized in machine learning, and/or data But that product role is always there Thats the clearest direction Ive found so far Right So its like starting small, and then over time, you just build the expertise and you increase the complexity.And scale Great Thank you very much for being here with us, sharing all those different use cases of AI and giving that talk Thank you very much GUY SAMUELS: No, thank you.