MR + AI: Machine Learning for Language in HoloLens & VR Apps
•
1 recomendación•1,747 vistas
Artificial Intelligence (AI) is disrupting every area of computing, and Mixed Reality is no exception. The combination of MR+AI is a powerful integration that opens new possibilities for both enterprise and entertainment scenarios. This session is an exploration of how language APIs in Microsoft Cognitive Services can be used to augment Mixed Reality experiences. We’ll explore the use of Natural Language Understanding to build advanced voice commands, translate languages from within MR environments, and we’ll also dive into the creation of intelligent assistants using Microsoft Bot Framework and use them as “brains” for interactive entities in Mixed Reality. In a world where speech is the primary form of input, using Machine Learning to process language input and understand the user’s intent is of paramount importance. Come learn how it’s done so you can build smart Mixed Reality experiences with the power to disrupt any industry.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a MR + AI: Machine Learning for Language in HoloLens & VR Apps
Similar a MR + AI: Machine Learning for Language in HoloLens & VR Apps (20)
Más de Nick Landry
Más de Nick Landry (20)
Último
Último (20)
MR + AI: Machine Learning for Language in HoloLens & VR Apps
- 1. Mixed Reality + AI: Machine Learning for Language in HoloLens & VR Apps Nick Landry Sr. Software Engineer, Commercial Software Engineering activenick@microsoft.com github.com/ActiveNick slideshare.net/ActiveNick @ActiveNick
- 3. AI Services in Mixed Reality Experiences
- 4. | MR Jam at Build AI, Machine Learning and Deep Learning 1950 1960 1970 1980 1990 2000 2010
- 5. | MR Jam at Build Azure AI Services Azure Infrastructure Tools Microsoft AI Platform
- 6. | MR Jam at Build MR+AI Services • Computer Vision + Holographic/AR • Language Services for MR Cognitive Services • Access to cloud data (SQL, Cosmos, etc.) from MR • Calling Azure ML APIs from MR Custom AI & Data Services • Smart assistants powered by Bots • Trained intelligent agents powered by ML Immersive Agents • Offline AI access via Windows ML • Access to Deep Learning frameworks (CNTK, TF, etc.) Local AI Services
- 7. | MR Jam at Build Microsoft Azure Cognitive Services https://www.microsoft.com/cognitive-services/ Emotion Speaker Recognition Speech Custom Recognition Computer Vision Face Video microsoft.com/cognitive Linguistic Analysis Language Understanding Bing Spell Check Entity Linking Knowledge Exploration Academic Knowledge Bing Image Search Bing Video Search Bing Web Search WebLM Text Analytics Recommendations Bing Autosuggest Bing News Search Translator Custom Vision
- 8. | MR Jam at Build Language Understanding LUIS • LUIS: A machine learning-based service to build natural language into apps, bots, and IoT devices. • Designed to identify valuable information in conversations, LUIS interprets user goals (intents) and distills valuable information from sentences (entities), for a high quality, nuanced language model. • Quickly create enterprise-ready, custom models that continuously improve. • MR+LUIS: Replace rigid voice commands with Natural Language Understanding (aka NLP, using Cognitive Services).
- 9. Demo Calling Language Understanding Services from Windows Mixed Reality Applications in Unity
- 10. | MR Jam at Build MR+Azure Open Source Projects in the Works • Mixed Reality + Azure Samples by CSE • https://github.com/Microsoft/mixedreality-azure-samples • Two feature branches in development: • feature/LUIS: Reference architecture to facilitate LUIS integration with MR projects built with Unity • Article series in progress available at aka.ms/mrluis • feature/LUIScache: LUIS Caching Service is a reusable sample that showcases how to cache results from LUIS • Help provide partial support for LUIS in voice-based applications running on mobile and Mixed Reality devices in poor connectivity areas • More samples, solutions and guidance to come • Follow @ActiveNick, @jbienz, @meulta on Twitter
- 11. Bot Framework
- 12. | MR Jam at Build What is a Bot? { Your Code } REST Endpoint Direct Line Protocol Conversational and Business Logic Canvas Aware Context Sensitive SDK Bot Builder SDK Platform Platform Services HTTP REST Endpoint AI Intelligent Tools
- 13. | MR Jam at Build Microsoft Bot Framework Channels O N E B O T M U L T I P L E C A N V A S E S
- 14. Merck Lab HoloBot MR+AI case Study
- 15. | MR Jam at Build Merck Pharmaceuticals The Problem: • Lab researchers struggling to access experiment documents • Lab experiment documents hosted in corporate databases • Difficult to access in the middle of an experiment, scientists held paper versions as they work: tedious, non-productive • Merck was exploring ways to provide hands-free, voice-activated document access
- 16. | MR Jam at Build Merck Lab HoloBot • Holographic bot designed to assist lab researchers in looking-up experiment documents • Merck and Microsoft built a new chatbot using Node.js and the Microsoft Bot Framework • Bot is accessed via a voice-controlled holographic interface on Microsoft HoloLens and supports natural language processing (NLP) via LUIS • Lab researchers can view experiment instructions hands-free, eliminating the need for printed documents
- 17. Video – Merck Lab HoloBot http://aka.ms/Merck
- 18. | MR Jam at Build
- 19. | MR Jam at Build Project Execution: 3-day hackfest • Merck adopted the Microsoft Bot Framework after they were introduced to the HoloBot open source project • Executed as a three-day hackfest in New Jersey at Merck • Merck team was already well-versed in Node.js • Customer added more customizations to HoloBot following the hackfest
- 20. | MR Jam at Build Key Technologies • HoloBot Open Source Project • Microsoft Bot Framework v3 • Language Understanding Intelligent Service (LUIS), part of Microsoft Cognitive Services • Azure App Service & Node.js • Microsoft HoloLens & Windows Mixed Reality • Unity Game Engine • Mixed Reality Toolkit for Unity (MRTK)
- 21. | MR Jam at Build Merck Lab HoloBot – Solution Architecture
- 22. | MR Jam at Build About HoloBot • Open Source Project by Nick Landry • https://github.com/ActiveNick/HoloBot • Custom Mixed Reality channel built with Direct Line API • Works with any bot built with MS Bot Framework (Node or .NET) • Designed for Unity 2017.2+, works with HoloLens and immersive VR headsets • 3rd place Hackathon Winner at OneWeek 2017 – Customer Category
- 23. Demo Building HoloBot with Unity, Windows Mixed Reality, Bot Framework & C#
- 24. | MR Jam at Build Viewing Lab Documents with HoloLens • Office 365 & PDF documents are launched via hyperlink in Monument Web Browser app • 2D app takes the user out of the HoloBot 3D view • 2D apps allow basic Scroll, Drag & Zoom gestures + voice commands • Exclusive to Merck project • Future version of HoloBot could support document browsing natively in 3D view
- 25. | MR Jam at Build Potential HoloBot Uses for Your Projects Simple reusable solution that easily demonstrates the potential of Mixed Reality combined with AI services and a cloud backend in Azure 1 HoloBot model can easily be replaced to match any company branded asset using custom textures or full 3D models 2 HoloBot can be integrated as a virtual assistant for any immersive/VR or holographic Mixed Reality experience, powered by Bot Framework 3 Beyond LUIS, bots can connect to more advanced Machine Learning models or data sources, allowing voice-activated touch-free access 4
- 26. | MR Jam at Build Call to Action • Don’t build offline Mixed Reality experiences, infuse AI into your apps! • Transition your Mixed Reality apps from rigid voice commands to natural language understanding with LUIS. • Leverage other Speech & Language Cognitive Services in MR apps, including Translation, Bing, Linguistic Analysis, Text Analytics, etc. • Build your own bots, connect them to your data, extend them to the VR & holographic world with HoloBot. • Provide feedback, file issues, contribute to our repos, share what you build. • Engage with us here at Build and throughout the year.
- 27. | MR Jam at Build Session resources • LUIS Demo: https://github.com/ActiveNick/MR_LUIS • Translator Demo: https://github.com/ActiveNick/MR_Translator • HoloBot project: https://github.com/ActiveNick/HoloBot • Merck Case Study • Microsoft Blog Article: http://aka.ms/Merck • Case Study Video: https://www.youtube.com/watch?v=4bJonmQnl_4
- 28. | MR Jam at Build Resources for MR+Cloud Developers • Getting Started with Windows Mixed Reality • https://aka.ms/mr • Mixed Reality + Cloud Resources on GitHub • https://github.com/Microsoft/mixedreality-azure-samples • https://github.com/Unity3dAzure • Azure for Game Developers • https://docs.microsoft.com/sandbox/gamedev • CSE EEE MR+Cloud Software Engineers on GitHub • https://github.com/ActiveNick • https://github.com/jbienzms • https://github.com/meulta
- 29. | MR Jam at Build Bot Framework Resources • Microsoft Bot Framework: https://dev.botframework.com/ • LUIS: https://www.luis.ai/ • Azure Bot Service: • https://azure.microsoft.com/services/bot-service/ • Bot Builder SDK v3 on GitHub: • https://github.com/Microsoft/BotBuilder • Bot Builder SDK v4 on GitHub: • https://github.com/Microsoft/botbuilder-dotnet • Bot Framework Blog: https://blog.botframework.com/ • Cortana Developer Center: http://developer.microsoft.com/cortana
- 30. Thank you! Questions? Nick Landry Senior Software Engineer, Microsoft activenick@microsoft.com github.com/ActiveNick slideshare.net/ActiveNick @ActiveNick
- 31. © 2018 Microsoft Corporation. All rights reserved. Microsoft, Windows, Microsoft Surface and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.