Preview Release: ONNC - Open Neural Network Compiler

Skymizer and BITMAIN together are pleased to announce the preview release of ONNC, Open Neural Network Compiler, which is a collection of open source, modular, and reusable compiler algorithms/toolchains targeted on deep learning accelerator (DLA). Based on ONNX format, ONNC transforms ONNX models into binary machine code for DLA ASICs. ONNC leverages the IR design of ONNX and provides effective algorithms to alleviate the overhead of data movement. Furthermore, ONNC has been proven to be practical on BITMAIN’s cutting edge AI ASICs.

Easy Backend Integration

ONNC is integrated with the LLVM bitcode runtime and compiler. If a DLA already supports the LLVM compiler, it can be connected to ONNC seamlessly. This helps most CPUs, GPUs, and DSPs ported to ONNC in a very short time.

On the other hand, if a DLA has unique computation features and is not compatible to LLVM, ONNC also provides a modular framework to speed up the compiler development. DLA vendors can quickly customize an ONNC backend from a so called “vanilla” backend, which already provides some necessary optimization algorithms.

Reusable Compiler Optimizations

Two of ONNC’s contributions are dividing the AI compilation into several clear phases and giving the corresponding APIs for algorithm development. There are five phases carefully defined, each of which is focused on a particular compiler problem: IR building, partitioning, scheduling, allocation, and code emitting. ONNC also provides a series of optimization algorithms ready for use. They are general and reusable, including tensor selection, tensor liveness analysis, linear scan local memory allocation, etc. ONNC’s pass manager is flexible and similar to LLVM’s. AI researchers and engineers who are familiar with LLVM can intuitively contribute their general or target-specific optimization algorithms to ONNC.

First Backend support for BITMAIN’s AI ASIC

Now we have BITMAIN’s upcoming AI ASIC as our first backend support, and most of ONNX models can be supported by ONNC. ONNC core team will continue to work with BITMAIN to optimize the complier for generating the most efficient code for BITMAIN’s AI ASIC.

Status and Future Work

Project ONNC is not mature yet. Still we have a lot to do and need the community together to make it better. In this preview release, we carefully designed the software architecture so as to simplify the future development and maintenance. The items we have done include:

1. Clear compilation phases and the corresponding APIs.

2. Well-defined and extensible intermediate representation (IR) of target-specific instructions.

3. A pass manager that supports automatic scheduling according to the dependency claimed by each pass.

4. A Sophon backend that supports BITMAIN AI ASIC.

5. Shim library such as ADT, Diagnostics, JSON, etc.

Project ONNC follows the “release early, release often” principle, so the next release should come soon by the end of August. We plan to finish the following items.

1. An x86 backend that enables the execution of AI models on an x86 machine. That backend should support both JIT and interpreter way of execution.

2. Improvements of memory allocation based on liveness analysis.



To keep up with the latest development or make suggestions, please join the ONNC mailing list. Also, please head over to the official website and GitHub for more information.