Example: stock market

Software Engineering for Machine Learning: A Case Study

Software Engineering for Machine Learning: A Case StudySaleema AmershiMicrosoft ResearchRedmond, WA BegelMicrosoft ResearchRedmond, WA BirdMicrosoft ResearchRedmond, WA DeLineMicrosoft ResearchRedmond, WA GallUniversity of ZurichZurich, KamarMicrosoft ResearchRedmond, WA NagappanMicrosoft ResearchRedmond, WA NushiMicrosoft ResearchRedmond, WA ZimmermannMicrosoft ResearchRedmond, WA Recent advances in Machine learning have stim-ulated widespread interest within the Information Technologysector on integrating AI capabilities into Software and goal has forced organizations to evolve their developmentprocesses. We report on a Study that we conducted on observingsoftware teams at Microsoft as they develop AI-based applica-tions.

For simplicity the view in Figure 1 is linear, however, machine learning workflows are highly non-linear and contain several feedback loops. For example, if engineers notice that there is a large distribution shift between the training data and …

Tags:

  Linear, Engineering, Machine, Software, Learning, Software engineering for machine learning

Information

Domain:

Source:

Link to this page:

Please notify us if you found a problem with this document:

Other abuse

Transcription of Software Engineering for Machine Learning: A Case Study

1 Software Engineering for Machine Learning: A Case StudySaleema AmershiMicrosoft ResearchRedmond, WA BegelMicrosoft ResearchRedmond, WA BirdMicrosoft ResearchRedmond, WA DeLineMicrosoft ResearchRedmond, WA GallUniversity of ZurichZurich, KamarMicrosoft ResearchRedmond, WA NagappanMicrosoft ResearchRedmond, WA NushiMicrosoft ResearchRedmond, WA ZimmermannMicrosoft ResearchRedmond, WA Recent advances in Machine learning have stim-ulated widespread interest within the Information Technologysector on integrating AI capabilities into Software and goal has forced organizations to evolve their developmentprocesses. We report on a Study that we conducted on observingsoftware teams at Microsoft as they develop AI-based applica-tions.

2 We consider a nine-stage workflow process informed byprior experiences developing AI applications ( , search andNLP) and data science tools ( application diagnostics and bugreporting). We found that various Microsoft teams have unitedthis workflow into preexisting, well-evolved, Agile-like softwareengineering processes, providing insights about several essentialengineering challenges that organizations may face in creatinglarge-scale AI solutions for the marketplace. We collected somebest practices from Microsoft teams to address these addition, we have identified three aspects of the AI domain thatmake it fundamentally different from prior Software applicationdomains: 1) discovering, managing, and versioning the dataneeded for Machine learning applications is much more complexand difficult than other types of Software Engineering , 2) modelcustomization and model reuse require very different skills thanare typically found in Software teams, and 3) AI componentsare more difficult to handle as distinct modules than traditionalsoftware components models may be entangled in complexways and experience non-monotonic error behavior.

3 We believethat the lessons learned by Microsoft teams will be valuable toother Terms AI, Software Engineering , process, dataI. INTRODUCTIONP ersonal computing. The Internet. The Web. Mobile com-puting. Cloud computing. Nary a decade goes by without adisruptive shift in the dominant application domain of thesoftware industry. Each shift brings with it new softwareengineering goals that spur Software organizations to evolvetheir development practices in order to address the novelaspects of the latest trend to hit the Software industry is aroundintegrating artificial intelligence (AI) capabilities based onadvances in Machine learning . AI broadly includes technolo-gies for reasoning, problem solving, planning, and learning ,among others.

4 Machine learning refers to statistical modelingtechniques that have powered recent excitement in the softwareand services marketplace. Microsoft product teams have usedmachine learning to create application suites such as BingSearch or the Cortana virtual assistant, as well as platformssuch as Microsoft Translator for real-time translation of text,voice, and video, Cognitive Services for vision, speech, andlanguage understanding for building interactive, conversationalagents, and the Azure AI platform to enable customers to buildtheir own Machine learning applications [1]. To create thesesoftware products, Microsoft has leveraged its preexistingcapabilities in AI and developed new areas of expertise acrossthe this paper, we describe a Study in which we learned howvarious Microsoft Software teams build Software applicationswith customer-focused AI features.

5 For that, Microsoft hasintegrated existing Agile Software Engineering processes withAI-specific workflows informed by prior experiences in devel-oping early AI and data science applications. In our Study , weasked Microsoft employees about how they worked throughthe growing challenges of daily Software development specificto AI, as well as the larger, more essential issues inherent in thedevelopment of large-scale AI infrastructure and teams across the company having differing amounts ofwork experience in AI, we observed that many issues reportedby newer teams dramatically drop in importance as the teamsmature, while some remain as essential to the practice of large-scale AI. We have made a first attempt to create a processmaturity metric to help teams identify how far they have comeon their journeys to building AI a key finding of our analyses, we discovered three funda-mental differences to building applications and platforms fortraining and fielding Machine - learning models than we haveseen in prior application domains.

6 First, Machine learning is allabout data. The amount of effort and rigor it takes to discover,source, manage, and version data is inherently more complexand different than doing the same with Software code. Second,building for customizability and extensibility of models requireteams to not only have Software Engineering skills but almostFig. 1. The nine stages of the Machine learning workflow. Some stages are data-oriented ( , collection, cleaning, and labeling) and others are model-oriented( , model requirements, feature Engineering , training, evaluation, deployment, and monitoring). There are many feedback loops in the workflow. The largerfeedback arrows denote that model evaluation and monitoring may loop back to any of the previous stages. The smaller feedback arrow illustrates that modeltraining may loop back to feature Engineering ( , in representation learning ).

7 Always require deep enough knowledge of Machine learning tobuild, evaluate, and tune models from scratch. Third, it can bemore difficult to maintain strict module boundaries betweenmachine learning components than for Software engineeringmodules. Machine learning models can be entangled incomplex ways that cause them to affect one another duringtraining and tuning, even if the Software teams building themintended for them to remain isolated from one lessons we identified via studies of a variety of teamsat Microsoft who have adapted their Software engineeringprocesses and practices to integrate Machine learning can helpother Software organizations embarking on their own pathstowards building AI applications and this paper, we offer the following ) A description of how several Microsoft Software en-gineering teams work cast into a nine-stage workflowfor integrating Machine learning into application andplatform )

8 A set of best practices for building applications andplatforms relying on Machine ) A custom Machine - learning process maturity model forassessing the progress of Software teams towards excel-lence in building AI ) A discussion of three fundamental differences in howsoftware Engineering applies to Machine - learning centriccomponents vs. previous application BACKGROUNDA. Software Engineering ProcessesThe changing application domain trends in the softwareindustry have influenced the evolution of the Software pro-cesses practiced by teams at Microsoft. For at least a decadeand a half, many teams have used feedback-intense Agilemethods to develop their Software [2], [3], [4] because theyneeded to be responsive at addressing changing customerneeds through faster development cycles.

9 Agile methods havebeen helpful at supporting further adaptation, for example,the most recent shift to re-organize numerous team s prac-tices around DevOps [5], which better matched the needsof building and supporting cloud computing applications change to DevOps occurred fairly quicklybecause these teams were able to leverage prior capabilities1 continuous integration and diagnostic-gathering, making itsimpler to implement continuous changes not only alter the day-to-day developmentpractices of a team, but also influence the roles that peopleplay. 15 years ago, many teams at Microsoft relied heavily ondevelopment triads consisting of a program manager (require-ments gathering and scheduling), a developer (programming),and a tester (testing) [6].

10 These teams adoption of DevOpscombined the roles of developer and tester and integratedthe roles of IT, operations, and diagnostics into the mainlinesoftware recent years, teams have increased their abilities to an-alyze diagnostics-based customer application behavior, prior-itize bugs, estimate failure rates, and understand performanceregressions through the addition of data scientists [7], [8], whohelped pioneer the integration of statistical and Machine learn-ing workflows into Software development processes. Somesoftware teams employpolymathdata scientists, who do itall, but as data science needs to scale up, their roles specializeinto domain experts who deeply understand the business prob-lems, modelers who develop predictive models, and platformbuilders who create the cloud-based ML WorkflowOne commonly used Machine learning workflow at Mi-crosoft has been depicted in various forms across industryand research [1], [9], [10], [11].


Related search queries