Crafting an optimal bespoke machine learning accelerator for an ASIC or FPGA implementation means balancing communication, computation, and data movement. Trading off the conflicting goals of performance, accuracy, and efficiency means building and evaluating multiple accelerator architectures, often in the context of a larger system. This talk will show how High-Level Synthesis can be used to quickly create and assess multiple RTL implementations for an AI accelerator from a single algorithmic description. We will explain how HLS can be used to find the optimal quantization for features and weights, layer-by-layer or globally for an entire network. And we will show how HLS can be used to investigate caching strategies and their impact on power and performance. High-Level Synthesis can be the key to deploying ML into the most constrained and challenging Edge and IoT systems.