AI Expo Africa 2021 ONLINE – Talk by Monica Livingston – Intel

hello everyone and thank you for joining me today i am so excited to be part of ai expo africa again and i really hope i get to see you all in person next year um i am monica livingston i am solutions and sales lead for ai at intel and today i want to talk to you about ai in production and by that i mean ai deployment at scale not specifically in the manufacturing space i know sometimes the word production is associated with the.

manufacturing space so i wanted to make sure that that i discussed that right off the bat so let me first address why this is an important topic ai and analytics are applicable everywhere where we have an influx of data available and that is essentially everywhere today so were getting more and more data from all of the sensors out there and while many companies are.

dabbling in ai and theyre experimenting most ai projects dont make it to scale deployment so my talk today is about the tools that are available to drive ai deployments at scale but first id like to start with examples of successful deployments a great example is dc water this is a leading utility based in washington dc and they were looking for a more efficient and a more accurate way to inspect their underground pipes for potential damage dc water turned to one of our partners.

wepro to co-develop a solution called pipe sleuth which uses deep learning and computer vision to automate the analysis of video footage from the pipes with the solution they were able to accelerate their time to ai while reducing the risk and the cost all while improving the reliability of their customers of their service to their customers so i like this example for several reasons first the usage case is.

universal water companies around the globe have similar challenges and second it speaks to the globalization and accessibility of ai solutions here we have a solution developed in india by wipro deployed in the united states and that can just as easily be deployed in africa or anywhere else in the world the next example um that ill talk about is another one with the global reach restaurants are everywhere um in this example burger king a global fast food leader has streamlined their development of a deep learning based drive-through food recommender using intels analytics zoo toolkit their goal was to improve.

their customer experience as the recommender was offering recommendations for what the customer might be interested in next as well as of course increase sales a customized recommendation was developed by considering two types of information first was the guest guest order behavior what the guest was actually ordering at the time and the second was context information like the weather the time and the location this recommender integrates spark data processing and distributed mx net training using ray in a unified pipeline on a single xeon cluster therefore eliminating the overhead of a heterogeneous solution and i like this example because the.

scale required demands cost optimizations and what i mean by that is the following when you have a proof of concept or an experiment you dont have as much constraint on the cost of the solution because the cost will be small no matter what and in the ai world what that essentially translates to is we dont have to worry about the infrastructure that we run this on because its a small deployment but because food fast food restaurants like burger king have hundreds of locations this ai solution needs to be deployed in.

hundreds of locations um so overspending on hardware would essentially multiply and ultimately make the solution cost prohibitive the roi just isnt there so the takeaway here is when you’re deploying at scale optimizing your software and benchmarking your hardware to make sure that you’re not over speccing that hardware is extremely important and one more example hi hi is an ai technology leader in china they are supplying medical.

institutions with high performance ai enabled medical imaging these solutions then are capable of successfully diagnosing several diseases cobit 19 being one of them um and this particular solution uses intel xeon scalable processes with deep learning boost as well as intel optimized software like openvino and the intel distribution of python and again the key here is using these optimizations in concert with your hardware infrastructure in order to make sure that you are optimizing your solution for cost so ive given you a couple of examples of ai deployment at scale and i wanted to start with those.

examples to show what the end state looks like not all companies are this far ahead on deploying ai but most companies expect that ai will transform their company and their industry within one through three years deloitte publishes a really great report called the state of ai in the enterprise this is publicly available and the chart shows that 61 of their respondents believe.

that ai will transform their industry within three years and 75 believe that ai will transform their organization within three years so lets look then at who is deploying ai it shouldnt be surprising that cloud service providers and digital services companies are in the best position to take advantage of ai they have available infrastructure and they have massive organizations for software as well as um as data so not only that but ai has enabled significant growth to their business a simple example is recommender systems which is one of the top three use cases for ai.

these companies also pioneer their own frameworks like google does with tensorflow and theyre actively acquiring ai startups and publishing extensively after cloud and digital services we see investment really across all other industries and of course these industries are different in size so ai investments may vary but we see meaningful deployments and business impact from ai-based applications across all segments so the point here is.

ai is transformative today it might be in its infancy it might be experiencing growing pains but it is impactful today so if at this point in our conversation you’re feeling overwhelmed that maybe you’re not adopting ai as quickly as it looks like the industrys adopting ai we should have the build versus buy conversation ai consumption in enterprise is in two categories the first one is ai embedded in enterprise software these are third-party applications that your company is already using for business processes like sas or sap and the second one is in-house projects built from the ground up.

now allow me to go back to deloitte and their survey for the additional data they have found that ai adopters tend to buy more than they tend to build seventeen percent um in fact say that they dont build an eai at all in house um and i should actually also mention that um the respondents of this survey um 100 of them are deploying ai so they did not look at the part of the market.

that that is not deploying ai all of these folks that are providing these these answers here have some level of ai deployment and then you can see that a significant part of the population 76 in fact both buy and build um depending on where they can gain a competitive differentiation and where existing solutions are already in the market so the main point here is.

you dont have to build ai solutions from the ground up in order for your business to take advantage of ai in fact most enterprise software companies are actively including ai into their solutions and for specific functions like recommenders like image like vision there are several companies in the market that already have ready-made customizable solutions and if you remember back to the example we discussed with dc water using the pipe slow solution from wepro that this is exactly what buy and customize is reaper already had a pipe.

solution that they were then able to customize um specifically for dc water okay so if you are taking the path of building some ai based solutions in your house you light in in-house you likely have run into some of these challenges and i want to spend just a little bit of time to acknowledge them here because they need to be addressed in order for an ai project to be successful and ill start with hype ai is such a buzzword today and everybody wants to be.

part of it but you dont need to solve every problem with ai make sure that ai is the best way to achieve your business outcome and that your return on investment meets your requirements business metrics still need to be met the second one is data so data is ex or ai is extremely data hungry and some rules of thumb for data needed to train a model are 12 months of data for forecasting and predictors and.

recommenders um if you’re doing anything with image you need about a thousand images per class um and so its important to have a broader data strategy beyond a single ai project centralizing your data removing silos making data accessible at the edge or wherever the insights are needed this should be part of a holistic company strategy and likely the largest cost adder to any ai project is the cost of moving data so.

moving data as little as possible within your solution is a game changer in terms of total cost of ownership now because ai models are dependent on all of this data and on regular retraining many business processes will need to change and to adapt you cant just put an ai model out there and expect that itll work forever and ever and ever.

they decay um there’s data drift um they need to continuously be retrained and this generally requires a cross-company buy-in so having an ai center of excellence or a champion for ai adoption within the company is generally very helpful your company also needs to keep up with new and changing regulations ethics liability process for the decisions that are made by by ai systems so generally having that ai center of excellence is extremely useful um and then there’s also a significant skills gap with respect to ai functions there’s data science ml engineering app.

developing schools are scrambling to develop these curriculums for ai and the good news is that a lot of these courses are available online for free there’s also several micro and nano degrees that are targeted specifically to these functions these are degrees that are available for these ai specialties and youve got companies like udacity or course era that are an easy path to upskilling.

so the questions that business leaders are asking and need to ask today is what expertise do we have available in house who do we need to hire and what upskilling is needed for our existing workforce does everybody ultimately need some level of understanding of of ai but heres some very positive news for for africa so i have this chart from slash data they did a report for for us on developer populations and.

they allowed me to use this chart um from their findings for for this specific webinar so whats really promising is to see that the developer population in africa is increasing quickly this data shows uh 14 growth over two quarters which means that education and training is accessible and that is extremely positive news that were not having these skill sets concentrated geographically as much so great great news for for africa here okay so in the second part of our chat.

today i want to talk about tools and ill be talking about tools that intel is making available but i do want to acknowledge that there are other tools out there so that you the customer the consumer has ample choice in the market and while this is getting a little bit deeper into the technical details um i promise i will make it clear why this is relevant.

to my business listeners today as well so before we get into tools i do want to address a question that i get asked every day and that is do i need new hardware for my ai solution and infrastructure is is a favorite soapbox of mine because there are so very few practitioners who understand how to optimize infrastructure for ai and there are very few systems engineers.

who understand the ai workloads and how to get performance from ai these are very very rare breeds out there and so people tend to throw the kitchen sink at an ai project they are accruing significant technical debt because theyre deploying more hardware than is needed and then of course you end up with underutilized resources and so this.

is one of the main reasons why pocs dont make it into production they just end up being too expensive when you do the math on the type of infrastructure that um that you’re being asked to purchase um so what my team spends a lot of time on is infrastructure optimization and model optimization with customers and the general rule of thumb is lets run your trained model and this is mainly an inference conversation because a lot of times training is a little bit more of a point solution.

but lets run your trained model on your existing infrastructure lets see what type of performance you get when you’re running in your normal environment along your usual set of workloads and if you’re running in the cloud benchmark across a different set of instances and from that you can assess whether the performance is good enough and whether you need more memory more io bandwidth more compute but always been benchmarking on your current environment.

sort of level sets um what potential incremental resources you might need so my second soapbox is optimizing software because the infrastructure is complex and you’re trading off really cost performance latency and bandwidth software is an extremely powerful lever weve seen performance improvements from 10x to 100x on the same hardware by just optimizing the software and intel offers an array of tools to do just that and by the way a majority of these are open source and free to use um so intel optimizes all layers of system software infrastructure from.

optimize from operation systems to applications this includes libraries industry frameworks and tools and the end goal is is simple offer tools to simplify and accelerate development of these end-to-end solutions as were deploying them at scale and as we talked about ais pervasive which also means more models are being taken into production this has given a rise to a high demand for ml ops time to solution or how long it takes to complete a project and ultimately deploy it is a key metric for companies and this includes sourcing labeling and processing the data and that can take.

several months experimenting with topologies tuning hyper parameters all of this is compute intensive but it could also be time intensive and then getting ready for inference at scale and integrating your model into a broader application because ai generally is just part of the application its not the entirety of it all of these steps are time consuming and when you’re also hit with the fact that there are disparate tools for all of these steps so you have the added complexity of having to make sure that.

you have a good process to output your data and model at whatever stage its at from one tool to another and so this is really what drives the need for ml ops and a unified platform for ai modeling and deployment and this is where that ai center of excellence that we talked about earlier really comes in handy if you have that ai center of excellence.

they can they can start standardizing and around some of these solutions for the entire company so you’re not reinventing the wheel every time um so there are several ml ops platforms that are emerging in the market lots of good choices um today id like to introduce you to one of those choices its called converge io and this is a company that intel acquired last year and continues to develop their end-to-end machine learning platform um they are um deploying a cloud and.

enterprise customers alike converge includes several intel tools and optimizations directly onto their platform which again is why theyre super interesting to me so all of the software that you see here on this slide is included on on the converge platform so um your developers have easy access to all of these tools in one spot and as new hardware is being launched like intels upcoming discrete graphics product line and the habana ai a6 that software too will be integrated directly.

into converge making it extremely easy to um to use that hardware and move models from one hardware to another if needed embedded into converge are several intel software solutions uh but these are also available in the intel one container portal so there’s definitely different ways to to consume um this these software offerings so lets talk a couple of them and and.

id like to do that in the context of um intel one api so one api is really two things first its an industry-wide initiative based on standards and open source um specification the second intel also has a product its a reference implementation the one api intel one api toolkits these are free to use um and so two things industry standards intel products so um the premise here that it is that its a unified language and libraries to deliver full native.

code performance across a range of hardware including cpus gpus fpgas and ai accelerators the premise here is that you write code once and you can deploy it on different hardware targets of course that requires some tuning and there are tools that come in one api for that specific tuning and optimization but the base the base model your code will be able to run on all of these um on all of these different um hardware architectures and so um intel has within one api has an ein analytics toolkit which includes.

many of the tools which which ive shown you on two slides ago remember i said you you can consume these a couple of different ways you can you can consume them and converge you can consume them individually through containers or you could use them as part of the ai and analytics toolkit and the primary benefit of course of one api is is speeding up development and deployment across architecture again the point here is that.

a lot of times these models move from training to inference to retraining across several different types of architectures and a lot of times its you know its maybe better to train on an accelerator but then deploy on base infrastructure like a cpu or maybe you want to train on on an accelerator and deploy on an fpga and so um the the benefits of this tool.

can really be realized in a lot of different ways so now i want to talk about a tool called analytics zoo and before i go on again i want to mention that intel 1 api openvino and analytics zoo are all free to use um so analytics zoo is a unified analytics and ei platform for distributed tensorflow keras and pytorch on apache spark and ray this allows us to scale to thousands of nodes and up to several terabytes of memory per node so remember the burger king example that we talked about at the beginning of our.

conversation i now want to take you into the kitchen so to speak to show you how their solution was achieved and for further reading you can navigate to the medium article that is linked at the bottom of this page for a lot more details on this specific solution and again this is this is all public information so the goal of this model is to recommend menu items to customers as theyre ordering so when a customer first pulls.

up their card is empty the recommender has to take into account things like time of day or weather or popular items in order to make the base recommendations because they dont know the customer they dont they dont have any information about them particularly now as items are added to the cart the recommender has to adjust real time and use this new information to decide what to recommend next so now to make a relevant recommendation the real time user behavior and context needs to be taken into account for example you dont want to recommend ice.

cream on a cold winter day or kids meals at midnight or soft drinks if there are already lots of other drinks in the cart so the challenge here was to make the model as context aware as possible as well as taking into account some real-time information like what the guest is is actually actively ordering so the model that was selected is called a transformer cross transformer or t cross t model this model can explain the sequence of each order going through a sequence.

transformer but it could also take constant context information into account through the context transformer and then both of these transformers are then combined to layton cross to generate recommendations um and so now what does this have to do with analytics zoo well the apis in analytics zoo to build all of these transformers the sequence transformer the context transformer and the t cross t transformer are available and theyre free um free to use and so this.

gives an example of just how much and how deep the availability is and the accessibility is for all of these ai resources now lets lets talk about this um example a little bit further in a traditional approach to building recommendation pipelines you would first set up two separate clusters one for the big data processing and you see that in the yarn cluster here in this diagram um all the way to the left and the other is dedicated to deep learning like again the the gpu cluster with mxnet in this picture the challenge here is that the structure.

introduces data transfer overhead and remember having to move data is very costly and you also have to manage two separate workflows and two systems in production to address these challenges um the solution that that the engineering team in this case came up with is they build a recommendation system on top of rayon spark in analytics zoo and this integrates spark data processing and distributed mxnet training into a unified pipeline that runs onto a singles young cluster this means that you are managing a.

unified workflow you’re not moving data as much because you’re running on a single cluster and you’re not spending as much money on infrastructure again single cluster so why does analytics do matter here um similar to my previous example rayon spark is open source and available on analytics zoo so you can literally fast track a recommender model and implementation by using free open source tools from analytics zoom and just for posterity lets look at how t-cross-t models perform when compared to other models um historical 12-month data was used remember when i told you.

in the past that hey about 12 months of data is what you need for for recommenders and predictors um what they did is they used 11 months for training and then the last month for validation so the goal of the model here in the offline training result is to predict the next best product for the guests to purchase and you can see that the t cross t model improves top one and top three accuracy over the baseline models.

and then in the in the a b testing result in a real world setting the t cross t model was run alongside google recommendation ai for for four weeks and what was evaluated was conversion rate and add-on sales and again you can see significant performance improvement by using the the t-cross-t model again i just wanted to share this so you guys can see the performance but also.

it is important to measure and have metrics in place for for your ai systems and do keep in mind that these systems do tend to degrade over time any ai models um decays over time and so you have to have a plan in place for maintaining that model and retraining and retesting so metrics are incredibly important in this space so there are um several reasons why i wanted to go into details about this burger king example and i.

realized it might have been a little bit too technical i apologize for that hopefully stayed with me so a lot of times examples are too high level and its difficult to envision how anyone can achieve and apply them and second i wanted to demonstrate the availability of open source free apis and tools that can really fast-track that ai based solution through development and deployment it doesnt mean you dont have to test it it doesnt mean it does it doesnt still have to meet all of your business.

metrics and processes but there’s a lot of tools out there that can help you fast track third because restaurants exist everywhere on earth food is universal i love food i wanted to demonstrate the universal applicability of these ai solutions and finally i want to drive home usability it doesnt matter what country you’re in and these tools and many more are available to you and accessible right now and so where do we go from here i think its.

its important to remember that ai is a destructive technology and it will require process changes within your company but you dont have to do everything in-house there are many off-the-shelf solutions that are likely to fit your needs and finally if you do build any house there are many free open source resources like analytics do like intel one api that are available for you to use.

um and finally please join um the other two intel sessions so safya boa will talk about intels vision for ai and his talk includes two very special guests and a very special announcement so please be sure to tune in or watch on demand and the other session is from walter riviera who will talk about federated learning now this is a really interesting topic on how institutions.

and by that think academia or hospitals can pull their data together to train a model um and and of course improve the model because its being trained on more data but without actually making that data visible to each other thereby preserving data privacy so two very cool sessions um for you to also check out and finally i would love to hear from you please feel free to connect with me and drop me a note and i thank you for your time i.

do hope to see you in person next year and i wish you a fantastic conference.