About the tinyML Summit

Designing new custom hardware accelerators for deep learning is clearly popular, but achieving state-of-the-art performance and efficiency with a new design is a complex and challenging problem. Innovation is required in both HW and SW domains, and we will be including topics from both today. This workshop will cover NVDLA HW’s design and methodology, leveraging domain-specific concepts to help achieve performance scalability as well as best-in-class computational efficiency. We will also be covering deep learning compiler concepts used to help convert NVDLA’s raw performance into accessible performance. By the completion of this workshop, attendees will be able to deploy their own NVDLA in the cloud and execute real-time inference with NVDLA’s open-source SW toolchain.

This workshop will show you how to run a magic wand and other machine learning examples in the TensorFlow Lite for Microcontrollers framework.

Advances in processing power and machine learning algorithms enable us to run machine learning models on tiny far edge devices. Arm’s latest improvements in SIMD and DSP extensions as well as our collaboration with Google TensorFlow Lite team is pushing machine smarts to our tiniest micro-controllers used in intelligent wireless sensors.

In this hands-on workshop, attendees will build a machine learning application with TensorFlow Lite Micro on Arm Cortex-M devices, then optimize our solution to unleash the unparalleled power of Arm microcontrollers.

Efficient deep learning computing requires algorithm and hardware co-design to enable specialization. However, the extra degree of freedom creates a much larger design space. Human engineers can hardly exhaust the design space by heuristics. We propose AutoML techniques to architect efficient neural networks. We investigate automatically designing small and fast models (ProxylessNAS), auto channel pruning (AMC), and auto mixed-precision quantization (HAQ). We demonstrate such learning-based, automated design achieves superior performance and efficiency than rule-based human design. Moreover, we shorten the design cycle by 200× than previous work, so that we can afford to design specialized neural network models for different hardware platforms. Finally, we accelerate computation-intensive AI applications including TSM for efficient video recognition and PVCNN for efficient 3D recognition on point clouds. Bio: Song Han is an assistant professor at MIT EECS. Dr. Han received the Ph.D. degree in Electrical Engineering from Stanford advised by Prof. Bill Dally. Dr. Han’s research focuses on efficient deep learning computing. He proposed “Deep Compression” and “ EIE Accelerator” that impacted the industry. His work received the best paper award in ICLR’16 and FPGA’17. He was the co-founder and chief scientist of DeePhi Tech which was acquired by Xilinx

Several critical applications require ML inference on resource-constrained devices, especially in the domain of Internet of Things like smartcity, smarthouse etc. Furthermore, many of these problems reduce to time-series classification. Unfortunately, existing techniques for time-series classification like recurrent neural networks are very difficult to deploy on the tiny devices due to computation and memory bottleneck. In this talk, we will discuss two new methods FastGRNN and SRNN that can enable time-series inference on devices as small as Arduino Uno that have 2KB of RAM. Our methods can provide as much as 70x speed-up and compression over state-of-the-art methods like LSTM, GRU, while also providing strong theoretical guarantees.

Development of audio TinyML algorithms and their implementation onto embedded hardware have enabled pre-ASIC study of the power-latency tradeoff of deep network architectures and feature representations. This talk presents algorithmic studies of TinyML deep networks on audio tasks including keyword spotting and speaker verification. We will compare the use of input features generated from the asynchronous events of a spiking audio sensor and input features generated from sampled audio. We will compare the throughput and energy efficiency of the FPGA implementations of recurrent neural network architectures used for different TinyML audio tasks. We will also discuss the system cost projection of implementing these networks onto an ASIC recurrent network accelerator.

MobileNets are used widely across industry and academia for mobile and embedded vision applications. In this talk I will present the background and technical details which lead to the design choices for MobileNet models including what makes them extremely efficient and well matched to the mobile use case. I will then present the newest MobileNetV3 models combining neural architecture search as well as new network design elements and their application to classification, object detection and semantic segmentation. I will conclude with results for adapting MobileNets to microcontrollers for IoT applications on the edge.

When TinyML methods and use-cases push on power, they inherently break new ground on cost, volume and ubiquitousness of smart systems. Innovations that democratize and proliferate intelligent behavior are consistent hallmarks of disruptive technology. We’ve invited a remarkable group of experienced technology investors to discuss the opportunities and pitfalls of new ventures building new products and business models around TinyML.

Modern day machine learning systems typically involve development of large networks/models that require deployment of huge cloud servers for the compute capacity which have big privacy implications and result in high costs. Small micro-controller powered devices are being overlooked for ML applications as achieving high model accuracies within the memory and latency constraints is a challenge. With the right set of tools and algorithms, this need not be the case. The technology presented here discusses ways to leverage the use of these devices that occupy our world in the billions and make them intelligent with light-weight machine learning. A platform has been developed that enables development of models optimized for these low power devices. This talk will discuss some exemplary commercial uses of this technology and the processes and suite of tools that are used to deliver new interactive experiences to hundreds of millions of end users.

All of the components necessary for making swarms of autonomous micro robots, or silicon insects, now exist. Like their biological inspiration, they will be able to scavenge and store energy, lift many times their own weight, sense and interact with their environment, and collaborate to achieve goals beyond the abilities of an individual. Two characteristics distinguish them from their biological cousins: communication, and intelligence. Silicon insects will use RF mesh networks which will enable them to communicate over distances and at bit rates orders of magnitude higher than real insects, and indeed to reach back to cloud-based resources unavailable to their chitinous friends. But their native intelligence will pale in comparison to the lowliest ant or laziest bee. Here we have an opportunity for TinyML.

With the recent advances in many areas of tiny ML several use cases where tiny ML is an absolute requirement has emerged. Hearing devices is an area where tiny ML holds the potential to radically transform the functionality of a 1 mW always on device. People with hearing problems can benefit from hearing device processing that separates competing voices into individual channels followed by resynthesizing of the auditory scene with spatial augmentation. The first successful segregation enhancement of competing voices required deep neural networks to achieve enough separation for the spatial augmentation to enhance segregation. It is furthermore a requirement that the latency of processing is below 20 ms – preferably less – and thus the processing must take place at the ear level without uplink and downlink latencies. Thus, for voice separation to work on the ears of people with hearing problems tiny ML is a necessity.

This talk presents the importance of Computer Vision and Deep learning techniques for Spatial computing and in specific for an Extended reality headset to be an effective spatial computing platform. The four fundamental perception modalities are introduced: head pose tracking, world reconstruction, eye tracking and hand tracking; emphasizing on the two main general themes: Understanding the world (spatial localization, environment mapping) and Understanding user’s intent (eye, gaze and hands). This talk will provide a deep dive into the main modalities along with key challenges, compute needs and open problems.

The artificial intelligence (AI) revolution is being widely spread even to the IoT with the help of 5G wireless communication. Compared to the Cloud-based or Edge-based AI applications, Internet-ofThings (IoT) applications require more autonomous, adaptive, and cooperative operations with extremely limited power, computing and memory resources without stable communication channels. AI, especially deep neural network (DNN), is the key technology to support such autonomy and adaptivity of the IoT machines in an unpredictable environment with limited available information. The IoT machines should contain not only inference but also training capabilities to adapt to environmental changes based on their experiences. Therefore, software and hardware co-optimization for DNN training is necessary for low-power and high-speed accelerators, in the same way it brought a dramatic increase in the performance of DNN inference accelerators. In addition, deep reinforcement learning (DRL) accelerators will be an essential part of the tide, showing a lot of benefits at making continuous decisions in an unknown environment, where labeled data is difficult to acquire.

This talk will show how Arm IP, including future processors based on Arm Helium technology, along with comprehensive software libraries and a widely supported ecosystem, provides new performance levels for system designers creating ML solutions of tomorrow. Combining signal processing and neural network acceleration brings multi-fold efficiency gains compared to existing microcontroller systems today. Coupled with enabling software, Arm IP enables mass deployment of next-generation AI platforms for IoT everywhere, within reach of every developer. We will discuss real-world examples and benchmarks to demonstrate the scalable performance of Arm system-on-chip (SoC) technologies to help you choose the right IP for your application. Attendees will walk away with an understanding of how to accelerate software development on a single Arm Cortex-M toolchain with optimized software libraries and the choice offered by the Arm ecosystem. All resulting in much more efficient, more secure solutions for AI in IoT that are easier to develop, deploy and maintain.

This Presentation describes an energy-efficient neural processing unit for battery-operated devices. The architecture utilizing threefold of parallelisms for computing Convolutional and Fully Connected layers to achieve object detection for the at-the far-edge-computation. In this presentation, we will present the underlying technology of co-designing neural net models and neural net accelerators to achieve the right tradeoff for the highest energy efficiency. This 128-MAC structure is capable of running a low-precision modified 2-bit Group-Net Network that can perform Image classification and accurate semantic segmentation of 23 frames per seconds while operating at one-half of one mWatt.

The design of energy-efficient hardware for edge AI revolves primarily around data movement and memory access. Especially for small models, it is not only feasible to have all memory on chip, but also reduce data movement further using in-memory compute cores. The community is currently investigating a variety of options in this space using compute-SRAM, Flash, RRAM, MRAM, phase change, and other emerging memory technologies. Will these ideas lead to a breakthrough in hardware efficiency? Will they deliver the programmability needed for practical deployment and real-world applications? A panel of experts from industry, academia and research institutes will debate these questions and spell out their future predictions for our audience.

ML Inference on embedded hardware has been attracting attention for many applications. Limitations on computing power, memory usage, and power consumption are major bottleneck to deploy deep neural networks on resource-constrained devices. Existing techniques to mitigate the limitations often involve power-latency tradeoff and yield degraded accuracy. This talk presents algorithm studies of deep model compression and computational acceleration on facial recognition tasks including face detection and anti-spoofing. A trainable quantizer is proposed to learn intervals to quantize activations and weights. This quantization-interval-learning allows the quantized networks to maintain the accuracy of the full precision (32-bit) networks with bit-width of activations and weights as low as 4-bit. The effectiveness of our trainable quantizer on ImageNet dataset will be demonstrated with various network architectures such as AlexNet, VGG-16, ResNet-50, Inception-V3, and so on. Furthermore, this talk proposes ‘On-sensor AI’ computing architecture while exploring the advantages of 4-bit (or less) operations of activations and weights. 4-bit MAC operation can be implemented by utilizing AND and COUNT operations only. The speed of computation of AND/COUNT implementation is similar to the one of Multiplier/Adder implementation. However, it becomes way faster if the bit-width goes down to 3-bit or less. Moreover, replacing the Multiplier & Adder with logical AND & COUNT operators can reduce the number of transistor counts by over 40 times. This ‘On-sensor AI’ is challenging but promising. This will pave the way to deploy highly accurate tiny ML models to things in everyday life.

Providing machine learning based algorithms that are able to run efficiently on Tiny devices like microcontrollers is already a challenge. Making it easy and affordable for data scientists and embedded software experts is a key step for market adoption of this technology. STMicroelectronics has developed tool STM32Cube.AI so that customers have an easy path to enable neural networks on any device across the broad STM32 microcontroller portfolio. The tool maps and runs pre-trained Neural Networks on STM32 microcontrollers and supports a wide range of popular deep-learning training tools like Keras, Tensor Flow Lite, Caffe… and ONNX format. It also takes advantage of quantization by supporting post-training quantization and quantized-aware learning models. The presentation will include the latest features of the STM32Cube.AI tool. To complement STM32Cube.AI, STMicroelectronics has developed software packages for quick and easy prototyping with end-to-end Audio, Motion and Vision examples. Audio and Motion use cases, such as human activity recognition and audio-scene classification, are running on STM32L4 ultra-low-power microcontrollers. Computer Vision examples, like food classification, are running on STM32H7 microcontrollers. Examples cover a wide range of options like quantized or float models and different memory configurations.

In this talk, I will explore some of the ways that we are working on improving model inference efficiency for tiny devices – where power, area, memory, compute resources are limited. I will present results for a few of these: compute scheduling optimization, model compression, quantized inference, and inmemory computing. Finally, I will discuss our plans for next research steps to further understand and develop the technology.

Tiny machine learning (ML) is poised to drive enormous growth within the IoT hardware and software industry. Measuring the performance of these rapidly proliferating systems, and comparing them in a meaningful way presents a considerable challenge; the complexity and dynamicity of the field obscure the measurement of progress and make embedded ML application and system design and deployment intractable. To foster more systematic development, while enabling innovation, a fair, replicable, and robust method of evaluating tinyML systems is required. A reliable and widely accepted tinyML benchmark is needed. To fulfill this need, tinyMLPerf is a community-driven effort to extend the scope of the existing MLPerf benchmark suite (mlperf.org) to include tinyML systems. With the broad support of over 75 member organizations, the tinyMLPerf group has begun the process of creating a benchmarking suite for tinyML systems. The talk presents the goals, objectives, and lessons learned (thus far), and welcomes others to join and contribute to tinyMLPerf.