Tensor Manipulation Unit: Reconfigurable Near-Memory Tensor Manipulation for High-Throughput AI SoC
Tensor Manipulation Unit: Reconfigurable Near-Memory Tensor Manipulation for High-Throughput AI SoC
Abstract
While recent advances in AI SoC design have focused heavily on accelerating tensor computation, the equally critical task of tensor manipulation—centered on high-volume data movement with minimal computation—remains underexplored. This work addresses that gap by introducing the Tensor Manipulation Unit (TMU): a reconfigurable, near-memory hardware block designed to execute data-movement-intensive (DMI) operators efficiently. TMU manipulates long datastreams in a memory-to-memory fashion using a RISC-inspired execution model and a unified addressing abstraction, enabling broad support for both coarse- and fine-grained tensor transformations. The proposed architecture integrates TMU alongside a TPU within a high-throughput AI SoC, leveraging double buffering and output forwarding to improve pipeline utilization. Fabricated in SMIC 40 nm technology, the TMU occupies only 0.019 mm2 while supporting over 10 representative TM operators. Benchmarking shows that TMU alone achieves up to 1413.43× and 8.54× operator-level latency reduction over ARM A72 and NVIDIA Jetson TX2, respectively. When integrated with the inhouse TPU, the complete system achieves a 34.6% reduction in end-to-end inference latency, demonstrating the effectiveness and scalability of reconfigurable tensor manipulation in modern AI SoCs.
Thank you for visiting this project page. Register this project to explore its novelty and implementation support.
