Food Image Recognition by Deep Learning - Nvidia

1 food Image Recognition by deep LearningAssoc. Prof. Steven HOIS chool of Information SystemsSingapore Management UniversityNational Day Rally 2017: Singapore's War on Four simple ways to fight diabetes: Go for regular medical check-ups; Exercise more; Watch your diet; and Cut down on soft drinks. -PM Lee Hsien LoongTraditional food food LoggingHealthy 365 Powered byRoadmapCasesResearchApproachProblemFoo d Image Recognition Visual Recognition Laksa?Machine LearningFood Image Recognition Could be very Tea or Teh Teh, tea with milk and sugar Teh-C, tea with evaporated milk Teh-C-kosong, tea with evaporated milk and no sugar Teh-O, tea with sugar only Teh-O-kosong, plain tea without milk or sugar Tehtarik, the Malay tea Teh-halia, tea with ginger water Teh-bing, tea with ice, akaTeh-ice Teh-siu-dai, tea with less sugar Teh-gah-dai, tea with extra sweetened Name HierarchyTeh O siudaiTeh O kosongTeh OTea, no milkGreen teaGreen tea ( no sugar)Iced lemon teaGreen teaIced lemon teaFood ItemVisual FoodFood Recognition Classical Computer Vision Pipeline deep Learning ApproachFeatureExtractionTrainable Classifier(ML)LaksaMeesiamMeeGorengFeatu reExtractionTrainable Classifier(ML)

2 LaksaMeesiamMeeGorengDeep NNDeep LearningDeep Convolutional Neural Networks (CNN) Convolutional Neural Networks (CNN)Photos taken form [LeCunet a. 1998] deep CNN for Visual Recognition Revolution of Depth From AlexNet(8-layers) in 2012[ Krizhevskyet al. 2012 ]Why deep Learning ?13 DeepLearningSmall dataBig dataData SizeAccuracyTraditional LearningProductDataMachine LearningHPC(GPU)GPU for High Performance Computing deep Learning on GPU Clusters DGX-1: Nvidia Pascal -powered Tesla P100 Performance equal to 250 conventional 1stDGX-1 deep Learning Supercomputer (with P100 GPUs) Nvidia DGX-1AI SupercomputerSG-FOODSGFOOD Data StatisticsSGFood724 DatasetTrainingValidationTest# total images361,6767,24036,200# Image per class~5001050 Histogram of #visual foods (724 visual food classes)# food Items:1038#Visual food : 724# food Category: 158 FoodAI.

3 Open API Services ArchitectureAPI ServiceMODELINFERENCE ENGINEDATABASEANNOTATIONSYSTEMMODELTRAIN INGEXTERNAL DATACOLLECTIONWebAppFrontendBackendOffli neRoadmapCasesResearchApproachProblemRes earch Challenges How to train a good CNN model? How to deal with new food ? How the labeled data size affects the accuracy?Model Training A Family of CNN models for visual Recognition An Analysis of deep Neural Network Models for Practical Applications Alfredo Canziani, Adam Paszke, Eugenio CulurcielloPublished 2016 in ArXivImageNet 1000 classes, million images for trainingExperimental Setups CNN Models GoogleNet ResNet: 18, 50, 101, 152 Settings Toolbox: Caffe& TensorFelow Finetunedfrom ImageNet pretrainedmodels Batch Size: From 16 to 128 Optimizer: SGD with momentum/RMS Prop/Adam Learning rate.

4 Fixed/multi-step/exponential decay Dropout/Batch NormalizationsBenchmark of FoodAIModels (SGFOOD)Top-1 Accuracy (%)Top-5 Accuracy (%) (IMAGENET)Top-1 Accuracy (%)Top-5 Accuracy (%) visual food classes, 361,676 images for training, ~500 images per class1000 object classes, million images for training, 1200 images per classFood Saliency MapHow to handle NEW food ? Too many possible food items in the market Only consider popular food for majority of users New food has few images available at the beginningNew food Discovery New food Image annotation Model Re-training with new foodUpdate FoodAIInference Engine What if only 10x less amount of labeled datais available to train an CNN model?Training on 10x less labeled O P-1 A C C U R A C YT O P-5 A C C U R A C YResNet-50 (10%)ResNet-50(10%)+augmentationResNet-5 0 (100%)RoadmapCasesResearchApproachProble mCase Studies: food logging photos from usersWebMobile AppPowered byCase Studies: Easy CasesKopi OAmericanoCase Studies: Hard CasesLarge inter-class similarity ( , drinks)Large inter-class similarity ( , drinks)Case Studies: Hard CasesPlain PorridgeSoya milkInstant CoffeeTehC / TehInstant CoffeeTehOTeh/ TehCCase Studies: Hard CasesLarge inter-class similarity ( , drinks)Large intra-class diversity ( , Economy rice)Case Studies: Hard CasesIncomplete FoodCase Studies: Hard CasesNon food Case Studies.

5 Hard CasesPoorly taken photos (illumination, rotation, occlusion, etc)Case Studies: Hard CasesMultiple food itemsCase Studies: Hard CasesUnknown food / food not in our listCase Studies: Hard CasesHow to build a more sustainable solution?CrowdsourcingBetter LearningGo beyond supervised CNNC ombined with human You!