AI Navigator: How to Implement AI model

Overview

This page briefly explains how to implement AI models converted using RUHMI and its inference process in your AI Application.
You need to follow the steps below:

  1. Copy RUHMI output files into the src directory
  2. Implement the AI Model and Inference
  3. Build & Run
Note This guide is based on the following references and summarizes the necessary steps for implementation on AI Navigator. Please refer to them for details.

1. Copy RUHMI output files into the src directory

RUHMI outputs the following files including the converted AI models.

Ethos
CPU
The RUHMI output files are as below. Please copy these files into your directory, excluding hal_entry.c.

When using Ethos-U (i.e., unchecking the option "Use only CPU" in the AI model conversion tool), hal_entry.c is generated.
This code runs the inference and compares the result with the dummy and actual output.
*If you want to run this hal_entry.c as is, you need to prepare the files for RTT Viewer. Go to this page to download the SEGGER_RTT folder.


.
├── ethosu_common.h
├── hal_entry.c     # HAL entry example
├── model.c     # AI Model file
├── model.h
├── model_io_data.c # AI Model I/O data file
├── model_io_data.h
├── sub_0000_command_stream.c   # Ethos-U55 subgraph generated C source code 
├── sub_0000_command_stream.h
├── sub_0000_invoke.c
├── sub_0000_invoke.h
├── sub_0000_io_data.c
├── sub_0000_io_data.h
├── sub_0000_model_data.c
├── sub_0000_model_data.h
├── sub_0000_tensors.c
├── sub_0000_tensors.h
└── ...

The RUHMI output files are as below.


.
├── compute_sub_0000.c  # CPU subgraph generated C source code, including inference process.
├── compute_sub_0000.h
├── ...
├── kernel_library_int.c    # kernel library if CPU subgraphs are present 
├── kernel_library_int.h
├── kernel_library_utils.c
├── kernel_library_utils.h
├── model_io_data.c  # AI Model I/O data file
└── model_io_data.h


2. Implement the AI Model and Inference

The following explains how to implement AI models and the inference process based on an AI application generated by RUHMI.

Click the button that matches your use case.

Ethos
CPU
Follow the steps below to implement AI inference using Ethos-U.
  1. Initialize Memory
  2. Initialize Ethos-U
  3. Add AI inference
  4. Stop Ethos-U
Reference： Runtime API - CPU + Ethos-U deployment

i. Initialize Memory

Initialize the memory used to store data such as the model.
The following is an example when using SDRAM. 
R_BSP_SdramInit(true);

ii. Initialize Ethos-U (NPU)

Initialize and start Ethos-U using RM_ETHOSU_Open() as shown below. 
/* Ethos-U Initialization */
int status = FSP_SUCCESS;
status = RM_ETHOSU_Open(&g_rm_ethosu0_ctrl, &g_rm_ethosu0_cfg);
if (status != FSP_SUCCESS) {
    /* Error Handling */
}

Tips If RM_ETHOSU_Open() fails, check the heap size setting.
Click "FSP Configuration" and select "BSP" tab. Then, in RA Common, enter an appropriate value in the "Heap size (bytes)" field.

iii. Add AI inference

You need to allocate the necessary memory for AI inference first, and then run it.
The inference process is executed by RunModel(), which is defined in model.h.
The pointer to the input data is defined as an int_8 model in model_io_data.h as below.

#include "model.h"

#include "model_io_data.h"

    ...
memcpy(GetModelInputPtr_input0(), model_input0, model_input_SIZE0);  
memcpy(GetModelInputPtr_input1(), model_input1, model_input_SIZE1);  
/* Run inference */
RunModel(false);

iV. Stop Ethos-U

After completing the entire inference process, stop Ethos-U with RM_ETHOSU_Close(). 
/* Stop Ethos */
status = RM_ETHOSU_Close(&g_rm_ethosu0_ctrl);
if (status != FSP_SUCCESS) {
    /* Error Handling */
}
Follow the steps below to implement AI inference using CPU only.
  1. Allocate the I/O buffers
  2. Allocate the intermediate buffer
  3. Run inference
Reference: Runtime API - CPU only deployment

i. Allocate the I/O buffers

Open compute_sub_0000.h and check the size of the input and output buffers.
/* File: compute_sub_0000.h */
void compute_sub_0000(
    /* buffer for intermediate results */
    uint8_t* main_storage, /* should provide at least <intermediate_buffers_size> bytes of storage */

    /* inputs */
    const int8_t <input_name>[XXX],
    /* outputs */
     int8_t <output_name>[YYY]
   );
Define the input and output buffers of the same size in your AI application code. 
/* Input buffer */
static int8_t input_buffer[XXX];
/* Output buffer */
static int8_t output_buffer[YYY];

ii. Allocate the iintermediate buffer

Reopen compute_sub_0000.h and check the size of the iintermediate buffer.
/* File: compute_sub_0000.h */
enum BufferSize_sub_0000 {
  kBufferSize_sub_0000 = <intermediate_buffers_size>
};
Define the iintermediate buffer of the same size in your AI application code. 
#include "compute_sub_0000.h"

...
/* Intermediate buffer */
static uint8_t compute_buffer[kBufferSize_sub_0000];

iii. Run inference

Run the inference with compute_sub_0000(). 
#include "compute_sub_0000.h"


/* Input buffer */
static int8_t input_buffer[XXX];
/* Output buffer */
static int8_t output_buffer[YYY];
/* Intermediate buffer */
static uint8_t compute_buffer[kBufferSize_sub_0000];

/* Run inference */
compute_sub_0000(compute_buffer, input_buffer, output_buffer);


3. Build & Run

Click “Build” in the AI Application view of AI Navigator to build your project.
Then, click “Run on the Board” in the AI Navigator menu and click “Run the AI”.

Wrapping up

That’s all for the explanation of how to implement the converted AI models into your project.
For more information, please refer to RUHMI GitHub.