---
title: "LAB05: Level 3 Exercises"
---

## Hardware Deployment Challenges

These exercises require physical hardware (Arduino Nano 33 BLE Sense or ESP32) and focus on real-world TensorFlow Lite deployment, memory optimization, and performance benchmarking.

::: {.callout-warning}
## Hardware Required
- Arduino Nano 33 BLE Sense OR ESP32-DevKitC
- USB cable for programming
- Arduino IDE 2.0+ with TensorFlow Lite Micro library
- Training environment with TensorFlow 2.x

You need the devices listed in the [Hardware Guide](../../resources/hardware.qmd) to complete these exercises.
:::

### Exercise 1: TFLite Model Deployment Pipeline
**Difficulty:** Level 3
**Hardware:** Arduino Nano 33 BLE Sense or ESP32
**Objective:** Deploy a quantized TensorFlow Lite model to hardware and verify inference accuracy.

**Task:**
Create a complete pipeline that trains a simple gesture classifier, converts it to TFLite format, and deploys it to your microcontroller.

**Steps:**
1. Train a small MLP model (32-16-4 architecture) for 4 gesture classes using synthetic accelerometer data
2. Convert the model to TensorFlow Lite with INT8 quantization
3. Generate a C array header file from the .tflite model
4. Deploy to Arduino/ESP32 and verify model loads successfully
5. Compare output predictions between Python and embedded versions

**Requirements:**
- Model must fit in <50KB flash memory
- Inference latency must be <10ms
- Quantized model accuracy within 5% of original

::: {.callout-tip collapse="true"}
## Solution

```python
# Part 1: Train and convert model (train_gesture_model.py)
import numpy as np
import tensorflow as tf
from tensorflow import keras

# Generate synthetic gesture data
def generate_gesture_data(num_samples=1000):
    """Generate 4 gesture classes from accelerometer patterns"""
    data = []
    labels = []

    for gesture_id in range(4):
        for _ in range(num_samples // 4):
            # Create characteristic patterns for each gesture
            if gesture_id == 0:  # Swipe left
                x = np.sin(np.linspace(0, 2*np.pi, 50)) + np.random.normal(0, 0.1, 50)
                y = np.random.normal(0, 0.2, 50)
                z = np.ones(50) + np.random.normal(0, 0.1, 50)
            elif gesture_id == 1:  # Swipe right
                x = -np.sin(np.linspace(0, 2*np.pi, 50)) + np.random.normal(0, 0.1, 50)
                y = np.random.normal(0, 0.2, 50)
                z = np.ones(50) + np.random.normal(0, 0.1, 50)
            elif gesture_id == 2:  # Shake
                x = np.random.normal(0, 0.5, 50)
                y = np.random.normal(0, 0.5, 50)
                z = np.ones(50) + np.random.normal(0, 0.3, 50)
            else:  # Still
                x = np.random.normal(0, 0.05, 50)
                y = np.random.normal(0, 0.05, 50)
                z = np.ones(50) + np.random.normal(0, 0.05, 50)

            # Flatten to single feature vector
            sample = np.concatenate([x, y, z])
            data.append(sample)
            labels.append(gesture_id)

    return np.array(data), np.array(labels)

# Generate training and test data
X_train, y_train = generate_gesture_data(800)
X_test, y_test = generate_gesture_data(200)

print(f"Training data shape: {X_train.shape}")
print(f"Input features: {X_train.shape[1]}")

# Build compact model
model = keras.Sequential([
    keras.layers.Input(shape=(150,)),
    keras.layers.Dense(32, activation='relu'),
    keras.layers.Dense(16, activation='relu'),
    keras.layers.Dense(4, activation='softmax')
])

model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

# Train model
history = model.fit(
    X_train, y_train,
    epochs=50,
    batch_size=32,
    validation_data=(X_test, y_test),
    verbose=1
)

# Evaluate
test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)
print(f"\nFloat32 model accuracy: {test_acc:.4f}")

# Convert to TensorFlow Lite with quantization
def representative_dataset():
    for i in range(100):
        yield [X_train[i:i+1].astype(np.float32)]

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8

tflite_model = converter.convert()

# Save model
with open('gesture_model.tflite', 'wb') as f:
    f.write(tflite_model)

print(f"\nModel size: {len(tflite_model)} bytes ({len(tflite_model)/1024:.2f} KB)")

# Verify quantized model accuracy
interpreter = tf.lite.Interpreter(model_content=tflite_model)
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

print(f"\nInput details: {input_details[0]}")
print(f"Output details: {output_details[0]}")

# Test quantized model
correct = 0
for i in range(len(X_test)):
    # Quantize input
    input_scale, input_zero_point = input_details[0]['quantization']
    input_data = X_test[i] / input_scale + input_zero_point
    input_data = input_data.astype(np.int8).reshape(1, 150)

    interpreter.set_tensor(input_details[0]['index'], input_data)
    interpreter.invoke()

    # Dequantize output
    output_data = interpreter.get_tensor(output_details[0]['index'])
    output_scale, output_zero_point = output_details[0]['quantization']
    output_data = (output_data.astype(np.float32) - output_zero_point) * output_scale

    pred = np.argmax(output_data)
    if pred == y_test[i]:
        correct += 1

quant_accuracy = correct / len(X_test)
print(f"Quantized model accuracy: {quant_accuracy:.4f}")
print(f"Accuracy drop: {(test_acc - quant_accuracy)*100:.2f}%")
```

```bash
# Part 2: Convert to C array
xxd -i gesture_model.tflite > gesture_model.h
```

```cpp
// Part 3: Arduino deployment (gesture_inference.ino)
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/micro/micro_log.h>
#include <tensorflow/lite/micro/system_setup.h>
#include <tensorflow/lite/schema/schema_generated.h>

#include "gesture_model.h"

// TFLite globals
namespace {
  const tflite::Model* model = nullptr;
  tflite::MicroInterpreter* interpreter = nullptr;
  TfLiteTensor* input = nullptr;
  TfLiteTensor* output = nullptr;

  // Tensor arena size - adjust based on model needs
  constexpr int kTensorArenaSize = 10 * 1024;
  uint8_t tensor_arena[kTensorArenaSize];
}

void setup() {
  Serial.begin(115200);
  while (!Serial);

  Serial.println("Gesture Recognition - TFLite Deployment");
  Serial.println("========================================");

  // Load model
  model = tflite::GetModel(gesture_model_tflite);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema version mismatch!");
    while(1);
  }
  Serial.println("Model loaded successfully");

  // Set up resolver
  static tflite::AllOpsResolver resolver;

  // Build interpreter
  static tflite::MicroInterpreter static_interpreter(
    model, resolver, tensor_arena, kTensorArenaSize);
  interpreter = &static_interpreter;

  // Allocate tensors
  TfLiteStatus allocate_status = interpreter->AllocateTensors();
  if (allocate_status != kTfLiteOk) {
    Serial.println("AllocateTensors() failed");
    while(1);
  }
  Serial.println("Tensors allocated");

  // Get input/output tensors
  input = interpreter->input(0);
  output = interpreter->output(0);

  // Print tensor info
  Serial.print("Input shape: [");
  for (int i = 0; i < input->dims->size; i++) {
    Serial.print(input->dims->data[i]);
    if (i < input->dims->size - 1) Serial.print(", ");
  }
  Serial.println("]");
  Serial.print("Input type: ");
  Serial.println(input->type);

  Serial.print("Output shape: [");
  for (int i = 0; i < output->dims->size; i++) {
    Serial.print(output->dims->data[i]);
    if (i < output->dims->size - 1) Serial.print(", ");
  }
  Serial.println("]");

  // Report memory usage
  Serial.print("Tensor arena used: ");
  Serial.print(interpreter->arena_used_bytes());
  Serial.print(" / ");
  Serial.print(kTensorArenaSize);
  Serial.println(" bytes");

  Serial.println("\nReady for inference!");
}

void loop() {
  // Generate test input (simulated sensor data)
  float test_data[150];
  for (int i = 0; i < 150; i++) {
    test_data[i] = sin(i * 0.1) + random(-10, 10) / 100.0;
  }

  // Quantize input
  float input_scale = input->params.scale;
  int input_zero_point = input->params.zero_point;

  for (int i = 0; i < 150; i++) {
    int8_t quantized = test_data[i] / input_scale + input_zero_point;
    input->data.int8[i] = quantized;
  }

  // Run inference
  unsigned long start_time = micros();
  TfLiteStatus invoke_status = interpreter->Invoke();
  unsigned long inference_time = micros() - start_time;

  if (invoke_status != kTfLiteOk) {
    Serial.println("Invoke failed!");
    return;
  }

  // Dequantize output
  float output_scale = output->params.scale;
  int output_zero_point = output->params.zero_point;

  Serial.print("Inference time: ");
  Serial.print(inference_time / 1000.0);
  Serial.println(" ms");

  Serial.print("Predictions: [");
  for (int i = 0; i < 4; i++) {
    float value = (output->data.int8[i] - output_zero_point) * output_scale;
    Serial.print(value, 4);
    if (i < 3) Serial.print(", ");
  }
  Serial.println("]");

  // Find max prediction
  int max_idx = 0;
  float max_val = -1000.0;
  for (int i = 0; i < 4; i++) {
    float value = (output->data.int8[i] - output_zero_point) * output_scale;
    if (value > max_val) {
      max_val = value;
      max_idx = i;
    }
  }

  const char* gestures[] = {"Swipe Left", "Swipe Right", "Shake", "Still"};
  Serial.print("Predicted gesture: ");
  Serial.println(gestures[max_idx]);
  Serial.println("---");

  delay(2000);
}
```

**Expected Outcomes:**
- Model file size: 10-20 KB
- Inference latency: 3-8 ms on Arduino Nano 33 BLE
- Tensor arena usage: 6-10 KB
- Quantized accuracy within 3-5% of Float32
- Serial output showing real-time predictions
:::

### Exercise 2: Tensor Arena Optimization
**Difficulty:** Level 3
**Hardware:** Arduino Nano 33 BLE Sense or ESP32
**Objective:** Minimize memory usage by optimizing tensor arena allocation.

**Task:**
Deploy a model and systematically reduce the tensor arena size to find the minimum required memory. Understand the relationship between model complexity and runtime memory needs.

**Steps:**
1. Deploy a CNN model for audio keyword spotting (or use the gesture model from Exercise 1)
2. Start with a large tensor arena (50 KB) and verify inference works
3. Use binary search to find the minimum viable arena size
4. Implement runtime memory reporting to track peak usage
5. Document the relationship between arena size and model architecture

**Requirements:**
- Implement automated arena size testing
- Report actual vs allocated memory
- Identify which layers consume most memory
- Achieve <20% memory overhead

::: {.callout-tip collapse="true"}
## Solution

```cpp
// memory_profiler.ino - Tensor Arena Optimization
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/micro/micro_log.h>
#include <tensorflow/lite/schema/schema_generated.h>

#include "gesture_model.h"

// Test different arena sizes
const int kTestArenaSizes[] = {50000, 25000, 15000, 10000, 8000, 6000, 5000, 4000};
const int kNumTests = sizeof(kTestArenaSizes) / sizeof(kTestArenaSizes[0]);

void testArenaSize(int arena_size) {
  Serial.print("\n========================================\n");
  Serial.print("Testing arena size: ");
  Serial.print(arena_size);
  Serial.println(" bytes");
  Serial.println("========================================");

  // Allocate tensor arena
  uint8_t* tensor_arena = new uint8_t[arena_size];
  if (tensor_arena == nullptr) {
    Serial.println("Failed to allocate arena!");
    return;
  }

  // Load model
  const tflite::Model* model = tflite::GetModel(gesture_model_tflite);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    delete[] tensor_arena;
    return;
  }

  // Setup resolver and interpreter
  static tflite::AllOpsResolver resolver;
  tflite::MicroInterpreter interpreter(
    model, resolver, tensor_arena, arena_size);

  // Try to allocate tensors
  TfLiteStatus allocate_status = interpreter.AllocateTensors();

  if (allocate_status != kTfLiteOk) {
    Serial.println("STATUS: FAILED - AllocateTensors() failed");
    Serial.println("Arena size too small!");
    delete[] tensor_arena;
    return;
  }

  Serial.println("STATUS: SUCCESS - Tensors allocated");

  // Get memory usage
  size_t used_bytes = interpreter.arena_used_bytes();
  float utilization = (used_bytes * 100.0) / arena_size;

  Serial.print("Arena allocated: ");
  Serial.print(arena_size);
  Serial.println(" bytes");

  Serial.print("Arena used: ");
  Serial.print(used_bytes);
  Serial.println(" bytes");

  Serial.print("Utilization: ");
  Serial.print(utilization, 1);
  Serial.println("%");

  Serial.print("Wasted space: ");
  Serial.print(arena_size - used_bytes);
  Serial.print(" bytes (");
  Serial.print(100.0 - utilization, 1);
  Serial.println("%)");

  // Test inference
  TfLiteTensor* input = interpreter.input(0);
  TfLiteTensor* output = interpreter.output(0);

  // Fill input with test data
  for (int i = 0; i < 150; i++) {
    input->data.int8[i] = random(-128, 127);
  }

  // Run inference
  unsigned long start = micros();
  TfLiteStatus invoke_status = interpreter.Invoke();
  unsigned long duration = micros() - start;

  if (invoke_status != kTfLiteOk) {
    Serial.println("Invoke() failed!");
  } else {
    Serial.print("Inference time: ");
    Serial.print(duration / 1000.0, 2);
    Serial.println(" ms");
    Serial.println("Inference: SUCCESS");
  }

  // Cleanup
  delete[] tensor_arena;
}

void findMinimumArena() {
  Serial.println("\n\nBINARY SEARCH FOR MINIMUM ARENA SIZE");
  Serial.println("=====================================\n");

  int low = 1000;   // Minimum possible
  int high = 50000; // Known working size
  int minimum_working = high;

  while (low <= high) {
    int mid = (low + high) / 2;

    Serial.print("Testing ");
    Serial.print(mid);
    Serial.print(" bytes... ");

    // Quick test
    uint8_t* tensor_arena = new uint8_t[mid];
    const tflite::Model* model = tflite::GetModel(gesture_model_tflite);
    static tflite::AllOpsResolver resolver;
    tflite::MicroInterpreter interpreter(model, resolver, tensor_arena, mid);

    TfLiteStatus status = interpreter.AllocateTensors();
    bool success = (status == kTfLiteOk);

    delete[] tensor_arena;

    if (success) {
      Serial.println("SUCCESS");
      minimum_working = mid;
      high = mid - 1;
    } else {
      Serial.println("FAILED");
      low = mid + 1;
    }

    delay(100);
  }

  Serial.print("\n*** MINIMUM ARENA SIZE: ");
  Serial.print(minimum_working);
  Serial.println(" bytes ***\n");

  // Test with 10% overhead for safety
  int recommended = minimum_working * 1.1;
  Serial.print("RECOMMENDED SIZE (with 10% overhead): ");
  Serial.print(recommended);
  Serial.println(" bytes");
}

void setup() {
  Serial.begin(115200);
  while (!Serial);

  Serial.println("TENSOR ARENA OPTIMIZATION TOOL");
  Serial.println("==============================\n");

  // Test predefined sizes
  for (int i = 0; i < kNumTests; i++) {
    testArenaSize(kTestArenaSizes[i]);
    delay(1000);
  }

  // Find minimum with binary search
  findMinimumArena();

  // Report available RAM
  Serial.println("\n\nSYSTEM MEMORY INFO");
  Serial.println("==================");

  #ifdef ARDUINO_ARCH_SAMD
    extern "C" char *sbrk(int i);
    char *ramstart = (char *)0x20000000;
    char *ramend = (char *)0x20008000;
    char *heapend = sbrk(0);
    int freeram = ramend - heapend;
    Serial.print("Free RAM: ");
    Serial.print(freeram);
    Serial.println(" bytes");
  #endif

  Serial.println("\nOptimization complete!");
}

void loop() {
  // Nothing in loop
}
```

**Expected Outcomes:**
- Discover minimum tensor arena size (typically 60-80% of initial guess)
- Understand memory layout: input tensors + output tensors + intermediate activations
- For gesture model: ~6-8 KB minimum arena
- For CNN models: 15-30 KB depending on architecture
- Binary search finds optimal size in 10-15 iterations
- 10% overhead recommended for stability

**Memory Breakdown Example:**
```
Input tensor (150 INT8):     150 bytes
Dense layer 1 activations:   32 bytes
Dense layer 2 activations:   16 bytes
Output tensor (4 INT8):      4 bytes
Working buffers:             ~5-6 KB
Total minimum:               ~6.2 KB
```
:::

### Exercise 3: Real-Time Performance Benchmarking
**Difficulty:** Level 3
**Hardware:** Arduino Nano 33 BLE Sense or ESP32
**Objective:** Profile model inference performance and identify bottlenecks.

**Task:**
Implement comprehensive performance monitoring for your deployed model, measuring inference latency, throughput, and layer-wise execution time.

**Steps:**
1. Deploy a model with timing instrumentation
2. Measure inference latency for 1000 consecutive inferences
3. Calculate statistics: mean, min, max, std dev, percentiles
4. Profile layer-wise execution time (if possible with available tools)
5. Test performance under different clock speeds and power modes

**Requirements:**
- Sub-millisecond timing resolution
- Statistical analysis of latency distribution
- Comparison with theoretical FLOPS calculations
- Power mode impact analysis (ESP32)

::: {.callout-tip collapse="true"}
## Solution

```cpp
// performance_benchmark.ino
#include <TensorFlowLite.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>

#include "gesture_model.h"

namespace {
  const tflite::Model* model = nullptr;
  tflite::MicroInterpreter* interpreter = nullptr;
  TfLiteTensor* input = nullptr;
  TfLiteTensor* output = nullptr;

  constexpr int kTensorArenaSize = 10 * 1024;
  uint8_t tensor_arena[kTensorArenaSize];

  // Benchmark parameters
  constexpr int kNumWarmupRuns = 10;
  constexpr int kNumBenchmarkRuns = 1000;

  // Latency storage
  float latencies[kNumBenchmarkRuns];
}

// Statistics calculation
struct Stats {
  float mean;
  float min;
  float max;
  float std_dev;
  float p50;  // Median
  float p95;
  float p99;
};

Stats calculateStats(float* data, int n) {
  Stats stats;

  // Sort data for percentiles
  float sorted[kNumBenchmarkRuns];
  memcpy(sorted, data, n * sizeof(float));

  // Bubble sort (simple for small arrays)
  for (int i = 0; i < n - 1; i++) {
    for (int j = 0; j < n - i - 1; j++) {
      if (sorted[j] > sorted[j + 1]) {
        float temp = sorted[j];
        sorted[j] = sorted[j + 1];
        sorted[j + 1] = temp;
      }
    }
  }

  // Calculate mean
  stats.mean = 0;
  for (int i = 0; i < n; i++) {
    stats.mean += data[i];
  }
  stats.mean /= n;

  // Min and max
  stats.min = sorted[0];
  stats.max = sorted[n - 1];

  // Standard deviation
  float variance = 0;
  for (int i = 0; i < n; i++) {
    float diff = data[i] - stats.mean;
    variance += diff * diff;
  }
  stats.std_dev = sqrt(variance / n);

  // Percentiles
  stats.p50 = sorted[n / 2];
  stats.p95 = sorted[(int)(n * 0.95)];
  stats.p99 = sorted[(int)(n * 0.99)];

  return stats;
}

void runBenchmark() {
  Serial.println("\n========================================");
  Serial.println("RUNNING PERFORMANCE BENCHMARK");
  Serial.println("========================================\n");

  // Warmup runs
  Serial.print("Warming up (");
  Serial.print(kNumWarmupRuns);
  Serial.println(" runs)...");

  for (int i = 0; i < kNumWarmupRuns; i++) {
    // Fill input with random data
    for (int j = 0; j < 150; j++) {
      input->data.int8[j] = random(-128, 127);
    }
    interpreter->Invoke();
  }

  Serial.println("Warmup complete!\n");

  // Benchmark runs
  Serial.print("Running benchmark (");
  Serial.print(kNumBenchmarkRuns);
  Serial.println(" inferences)...");

  unsigned long total_start = micros();

  for (int i = 0; i < kNumBenchmarkRuns; i++) {
    // Generate input data
    for (int j = 0; j < 150; j++) {
      input->data.int8[j] = random(-128, 127);
    }

    // Time single inference
    unsigned long start = micros();
    interpreter->Invoke();
    unsigned long duration = micros() - start;

    latencies[i] = duration / 1000.0;  // Convert to milliseconds

    // Progress indicator
    if ((i + 1) % 100 == 0) {
      Serial.print(".");
    }
  }

  unsigned long total_duration = micros() - total_start;

  Serial.println("\nBenchmark complete!\n");

  // Calculate statistics
  Stats stats = calculateStats(latencies, kNumBenchmarkRuns);

  // Print results
  Serial.println("========================================");
  Serial.println("BENCHMARK RESULTS");
  Serial.println("========================================\n");

  Serial.println("LATENCY STATISTICS:");
  Serial.print("  Mean:   ");
  Serial.print(stats.mean, 3);
  Serial.println(" ms");

  Serial.print("  Median: ");
  Serial.print(stats.p50, 3);
  Serial.println(" ms");

  Serial.print("  Min:    ");
  Serial.print(stats.min, 3);
  Serial.println(" ms");

  Serial.print("  Max:    ");
  Serial.print(stats.max, 3);
  Serial.println(" ms");

  Serial.print("  Std Dev: ");
  Serial.print(stats.std_dev, 3);
  Serial.println(" ms");

  Serial.print("  P95:    ");
  Serial.print(stats.p95, 3);
  Serial.println(" ms");

  Serial.print("  P99:    ");
  Serial.print(stats.p99, 3);
  Serial.println(" ms");

  Serial.println("\nTHROUGHPUT:");
  float throughput = (kNumBenchmarkRuns * 1000000.0) / total_duration;
  Serial.print("  Inferences/sec: ");
  Serial.println(throughput, 2);

  float total_time_sec = total_duration / 1000000.0;
  Serial.print("  Total time: ");
  Serial.print(total_time_sec, 3);
  Serial.println(" seconds");

  // Calculate theoretical operations
  // Model: 150->32->16->4
  // Operations: (150*32 + 32) + (32*16 + 16) + (16*4 + 4) = 4800 + 544 + 68 = 5412
  int total_ops = (150 * 32 + 32) + (32 * 16 + 16) + (16 * 4 + 4);
  float ops_per_sec = total_ops * throughput;

  Serial.println("\nCOMPUTATIONAL PERFORMANCE:");
  Serial.print("  Operations per inference: ");
  Serial.println(total_ops);

  Serial.print("  Operations/sec: ");
  Serial.print(ops_per_sec / 1000000.0, 2);
  Serial.println(" MOPS");

  // Memory info
  Serial.println("\nMEMORY USAGE:");
  Serial.print("  Tensor arena: ");
  Serial.print(interpreter->arena_used_bytes());
  Serial.print(" / ");
  Serial.print(kTensorArenaSize);
  Serial.println(" bytes");

  float utilization = (interpreter->arena_used_bytes() * 100.0) / kTensorArenaSize;
  Serial.print("  Utilization: ");
  Serial.print(utilization, 1);
  Serial.println("%");

  Serial.println("\n========================================\n");
}

void setup() {
  Serial.begin(115200);
  while (!Serial);

  Serial.println("TENSORFLOW LITE MICRO - PERFORMANCE BENCHMARK");
  Serial.println("==============================================\n");

  // Model setup
  model = tflite::GetModel(gesture_model_tflite);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while(1);
  }

  static tflite::AllOpsResolver resolver;
  static tflite::MicroInterpreter static_interpreter(
    model, resolver, tensor_arena, kTensorArenaSize);
  interpreter = &static_interpreter;

  if (interpreter->AllocateTensors() != kTfLiteOk) {
    Serial.println("AllocateTensors() failed!");
    while(1);
  }

  input = interpreter->input(0);
  output = interpreter->output(0);

  Serial.println("Model loaded successfully");
  Serial.print("Model size: ");
  Serial.print(gesture_model_tflite_len);
  Serial.println(" bytes");

  // Run benchmark
  delay(2000);
  runBenchmark();

  Serial.println("Benchmark suite complete!");
}

void loop() {
  // Nothing in loop
}
```

**Expected Outcomes:**

**Arduino Nano 33 BLE (64 MHz Cortex-M4):**
- Mean latency: 3-5 ms
- Throughput: 200-300 inferences/sec
- Very consistent timing (low std dev)
- ~5-10 MOPS performance

**ESP32 (240 MHz Xtensa):**
- Mean latency: 1-2 ms
- Throughput: 500-800 inferences/sec
- Slightly higher variance than Arduino
- ~15-25 MOPS performance

**Latency Distribution:**
- P50 very close to mean (stable performance)
- P99 typically 1.2-1.5× mean (cache effects)
- Max can show occasional spikes (interrupts)

**Optimization Tips:**
- Disable Serial printing during inference
- Use dedicated core on ESP32
- Disable WiFi/BLE during benchmarking
- Run at maximum CPU frequency
:::

### Exercise 4: Multi-Model Deployment Strategy
**Difficulty:** Level 3
**Hardware:** ESP32 (required for sufficient flash)
**Objective:** Deploy multiple TFLite models and implement dynamic model switching.

**Task:**
Deploy 3 different models to a single ESP32 and implement a system that can switch between them based on runtime conditions (e.g., battery level, accuracy requirements, input type).

**Steps:**
1. Create 3 models with different complexity levels: tiny (2KB), small (10KB), large (40KB)
2. Convert all to TFLite and generate C arrays
3. Implement a model manager that can load/unload models dynamically
4. Create a selection policy based on available memory and latency requirements
5. Benchmark switching overhead and memory usage

**Requirements:**
- All models must fit in ESP32 flash (4MB)
- Model switching must complete in <100ms
- Memory must be properly freed when switching
- Implement graceful degradation strategy

::: {.callout-tip collapse="true"}
## Solution

```python
# create_multi_models.py - Generate 3 models of different sizes
import numpy as np
import tensorflow as tf
from tensorflow import keras

def create_and_save_model(name, architecture, output_name):
    """Create model with specified architecture"""
    model = keras.Sequential([keras.layers.Input(shape=(150,))])

    for units in architecture:
        model.add(keras.layers.Dense(units, activation='relu'))

    model.add(keras.layers.Dense(4, activation='softmax'))

    # Dummy training
    X = np.random.randn(100, 150).astype(np.float32)
    y = np.random.randint(0, 4, 100)
    model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
    model.fit(X, y, epochs=5, verbose=0)

    # Convert to TFLite
    converter = tf.lite.TFLiteConverter.from_keras_model(model)
    converter.optimizations = [tf.lite.Optimize.DEFAULT]
    tflite_model = converter.convert()

    # Save
    with open(f'{output_name}.tflite', 'wb') as f:
        f.write(tflite_model)

    print(f"{name}: {len(tflite_model)} bytes, architecture: {architecture}")
    return len(tflite_model)

# Create three models
tiny_size = create_and_save_model("Tiny", [8, 4], "model_tiny")
small_size = create_and_save_model("Small", [32, 16], "model_small")
large_size = create_and_save_model("Large", [64, 32, 16], "model_large")

print(f"\nTotal size: {tiny_size + small_size + large_size} bytes")
```

```bash
# Convert to C arrays
xxd -i model_tiny.tflite > model_tiny.h
xxd -i model_small.tflite > model_small.h
xxd -i model_large.tflite > model_large.h
```

```cpp
// multi_model_manager.ino - ESP32 only
#include <TensorFlowLite_ESP32.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>

#include "model_tiny.h"
#include "model_small.h"
#include "model_large.h"

enum ModelSize {
  MODEL_TINY,
  MODEL_SMALL,
  MODEL_LARGE
};

class ModelManager {
private:
  const tflite::Model* current_model;
  tflite::MicroInterpreter* interpreter;
  uint8_t* tensor_arena;
  int arena_size;
  ModelSize active_model;
  tflite::AllOpsResolver resolver;

public:
  ModelManager() : current_model(nullptr), interpreter(nullptr),
                   tensor_arena(nullptr), arena_size(0) {}

  ~ModelManager() {
    cleanup();
  }

  void cleanup() {
    if (interpreter != nullptr) {
      delete interpreter;
      interpreter = nullptr;
    }
    if (tensor_arena != nullptr) {
      free(tensor_arena);
      tensor_arena = nullptr;
    }
    current_model = nullptr;
  }

  bool loadModel(ModelSize size) {
    unsigned long start_time = millis();

    Serial.print("Loading model: ");

    // Cleanup previous model
    cleanup();

    // Load new model
    const unsigned char* model_data;
    unsigned int model_len;

    switch (size) {
      case MODEL_TINY:
        Serial.print("TINY... ");
        model_data = model_tiny_tflite;
        model_len = model_tiny_tflite_len;
        arena_size = 5 * 1024;
        break;

      case MODEL_SMALL:
        Serial.print("SMALL... ");
        model_data = model_small_tflite;
        model_len = model_small_tflite_len;
        arena_size = 10 * 1024;
        break;

      case MODEL_LARGE:
        Serial.print("LARGE... ");
        model_data = model_large_tflite;
        model_len = model_large_tflite_len;
        arena_size = 20 * 1024;
        break;

      default:
        Serial.println("INVALID MODEL!");
        return false;
    }

    current_model = tflite::GetModel(model_data);
    if (current_model->version() != TFLITE_SCHEMA_VERSION) {
      Serial.println("Schema mismatch!");
      return false;
    }

    // Allocate tensor arena
    tensor_arena = (uint8_t*)malloc(arena_size);
    if (tensor_arena == nullptr) {
      Serial.println("Arena allocation failed!");
      return false;
    }

    // Create interpreter
    interpreter = new tflite::MicroInterpreter(
      current_model, resolver, tensor_arena, arena_size);

    if (interpreter->AllocateTensors() != kTfLiteOk) {
      Serial.println("Tensor allocation failed!");
      cleanup();
      return false;
    }

    active_model = size;

    unsigned long load_time = millis() - start_time;

    Serial.print("SUCCESS (");
    Serial.print(load_time);
    Serial.println(" ms)");

    Serial.print("  Model size: ");
    Serial.print(model_len);
    Serial.println(" bytes");

    Serial.print("  Arena used: ");
    Serial.print(interpreter->arena_used_bytes());
    Serial.print(" / ");
    Serial.print(arena_size);
    Serial.println(" bytes");

    return true;
  }

  float runInference(float* input_data, int input_size) {
    if (interpreter == nullptr) {
      Serial.println("No model loaded!");
      return -1;
    }

    TfLiteTensor* input = interpreter->input(0);

    // Copy input (assuming float model for simplicity)
    for (int i = 0; i < input_size && i < 150; i++) {
      input->data.f[i] = input_data[i];
    }

    unsigned long start = micros();
    interpreter->Invoke();
    unsigned long duration = micros() - start;

    return duration / 1000.0;  // Return latency in ms
  }

  ModelSize getActiveModel() {
    return active_model;
  }

  int getMemoryUsage() {
    return interpreter ? interpreter->arena_used_bytes() : 0;
  }
};

// Global model manager
ModelManager modelMgr;

// Model selection policy
ModelSize selectModel(int battery_level, bool high_accuracy_needed, int free_memory_kb) {
  Serial.println("\n--- Model Selection Policy ---");
  Serial.print("Battery level: ");
  Serial.print(battery_level);
  Serial.println("%");
  Serial.print("High accuracy needed: ");
  Serial.println(high_accuracy_needed ? "YES" : "NO");
  Serial.print("Free memory: ");
  Serial.print(free_memory_kb);
  Serial.println(" KB");

  // Policy rules:
  // 1. If battery low (<20%), use tiny model
  // 2. If high accuracy needed AND enough memory, use large
  // 3. Otherwise use small

  if (battery_level < 20) {
    Serial.println("Decision: TINY (battery saving mode)");
    return MODEL_TINY;
  }

  if (high_accuracy_needed && free_memory_kb > 30) {
    Serial.println("Decision: LARGE (high accuracy mode)");
    return MODEL_LARGE;
  }

  Serial.println("Decision: SMALL (balanced mode)");
  return MODEL_SMALL;
}

void demonstrateModelSwitching() {
  Serial.println("\n========================================");
  Serial.println("MODEL SWITCHING DEMONSTRATION");
  Serial.println("========================================\n");

  // Test data
  float test_input[150];
  for (int i = 0; i < 150; i++) {
    test_input[i] = sin(i * 0.1);
  }

  // Scenario 1: Normal operation
  Serial.println("\nSCENARIO 1: Normal operation");
  ModelSize model1 = selectModel(80, false, 100);
  modelMgr.loadModel(model1);
  float latency1 = modelMgr.runInference(test_input, 150);
  Serial.print("Inference latency: ");
  Serial.print(latency1, 2);
  Serial.println(" ms");

  delay(2000);

  // Scenario 2: Low battery
  Serial.println("\n\nSCENARIO 2: Low battery");
  ModelSize model2 = selectModel(15, false, 100);
  modelMgr.loadModel(model2);
  float latency2 = modelMgr.runInference(test_input, 150);
  Serial.print("Inference latency: ");
  Serial.print(latency2, 2);
  Serial.println(" ms");

  delay(2000);

  // Scenario 3: High accuracy needed
  Serial.println("\n\nSCENARIO 3: High accuracy needed");
  ModelSize model3 = selectModel(90, true, 100);
  modelMgr.loadModel(model3);
  float latency3 = modelMgr.runInference(test_input, 150);
  Serial.print("Inference latency: ");
  Serial.print(latency3, 2);
  Serial.println(" ms");

  // Summary
  Serial.println("\n\n========================================");
  Serial.println("SWITCHING SUMMARY");
  Serial.println("========================================");
  Serial.println("Scenario 1 (Normal):   SMALL model");
  Serial.println("Scenario 2 (Low batt): TINY model");
  Serial.println("Scenario 3 (Hi-acc):   LARGE model");
  Serial.println("\nGraceful degradation strategy:");
  Serial.println("- Battery < 20%: Force tiny model");
  Serial.println("- Memory < 30KB: Avoid large model");
  Serial.println("- Default: Small model (balanced)");
}

void setup() {
  Serial.begin(115200);
  while (!Serial);

  Serial.println("MULTI-MODEL DEPLOYMENT SYSTEM");
  Serial.println("==============================\n");

  Serial.println("Available models:");
  Serial.print("  TINY:  ");
  Serial.print(model_tiny_tflite_len);
  Serial.println(" bytes");

  Serial.print("  SMALL: ");
  Serial.print(model_small_tflite_len);
  Serial.println(" bytes");

  Serial.print("  LARGE: ");
  Serial.print(model_large_tflite_len);
  Serial.println(" bytes");

  Serial.print("\nTotal: ");
  Serial.print(model_tiny_tflite_len + model_small_tflite_len + model_large_tflite_len);
  Serial.println(" bytes");

  delay(2000);

  demonstrateModelSwitching();

  Serial.println("\n\nDemo complete!");
}

void loop() {
  // Nothing
}
```

**Expected Outcomes:**
- Model switching overhead: 50-100 ms
- Memory properly freed between switches
- Latency varies by model:
  - Tiny: ~1 ms, 2KB
  - Small: ~3 ms, 10KB
  - Large: ~8 ms, 40KB
- Policy enables adaptive performance/power trade-offs
- Graceful degradation under resource constraints
:::

### Exercise 5: Continuous On-Device Learning
**Difficulty:** Level 3
**Hardware:** ESP32 (4MB flash recommended)
**Objective:** Implement simple on-device model adaptation using transfer learning or fine-tuning.

**Task:**
Deploy a pre-trained model and implement a mechanism to collect new labeled samples on-device, then periodically update the final layer weights using simple gradient descent.

**Steps:**
1. Deploy a frozen feature extractor (all layers except last)
2. Implement a data collection buffer for new labeled samples
3. Implement simple SGD to update final layer weights
4. Validate updated model on holdout test set
5. Compare performance before and after adaptation

**Requirements:**
- Store at least 100 samples in EEPROM/Flash
- Implement mini-batch gradient descent
- Learning rate schedule (start at 0.01, decay)
- Measure accuracy improvement

::: {.callout-tip collapse="true"}
## Solution

```cpp
// on_device_learning.ino - ESP32 required
#include <TensorFlowLite_ESP32.h>
#include <tensorflow/lite/micro/all_ops_resolver.h>
#include <tensorflow/lite/micro/micro_interpreter.h>
#include <tensorflow/lite/schema/schema_generated.h>
#include <Preferences.h>

#include "model_small.h"  // Pre-trained model

// Training data storage
#define MAX_SAMPLES 100
#define INPUT_SIZE 150
#define OUTPUT_SIZE 4

struct TrainingSample {
  float input[INPUT_SIZE];
  int label;
};

TrainingSample training_buffer[MAX_SAMPLES];
int num_samples = 0;

// Model components
const tflite::Model* model = nullptr;
tflite::MicroInterpreter* interpreter = nullptr;
constexpr int kTensorArenaSize = 15 * 1024;
uint8_t tensor_arena[kTensorArenaSize];

// Last layer weights (16 inputs × 4 outputs = 64 weights + 4 biases)
float final_layer_weights[16][4];
float final_layer_biases[4];

// Learning parameters
float learning_rate = 0.01;
const float lr_decay = 0.95;
const int batch_size = 10;

Preferences preferences;

void setup() {
  Serial.begin(115200);
  while (!Serial);

  Serial.println("ON-DEVICE LEARNING DEMO");
  Serial.println("=======================\n");

  // Load model
  model = tflite::GetModel(model_small_tflite);
  if (model->version() != TFLITE_SCHEMA_VERSION) {
    Serial.println("Model schema mismatch!");
    while(1);
  }

  static tflite::AllOpsResolver resolver;
  static tflite::MicroInterpreter static_interpreter(
    model, resolver, tensor_arena, kTensorArenaSize);
  interpreter = &static_interpreter;

  if (interpreter->AllocateTensors() != kTfLiteOk) {
    Serial.println("AllocateTensors() failed!");
    while(1);
  }

  Serial.println("Model loaded successfully\n");

  // Initialize final layer weights randomly
  initializeWeights();

  // Load any previously saved samples
  loadTrainingData();

  // Generate some synthetic training data
  collectTrainingSamples(50);

  // Evaluate before training
  Serial.println("\n--- Before Training ---");
  float accuracy_before = evaluateModel();
  Serial.print("Accuracy: ");
  Serial.print(accuracy_before * 100, 1);
  Serial.println("%");

  // Train
  trainModel(20);  // 20 epochs

  // Evaluate after training
  Serial.println("\n--- After Training ---");
  float accuracy_after = evaluateModel();
  Serial.print("Accuracy: ");
  Serial.print(accuracy_after * 100, 1);
  Serial.println("%");

  Serial.print("\nImprovement: +");
  Serial.print((accuracy_after - accuracy_before) * 100, 1);
  Serial.println("%");

  // Save updated weights
  saveWeights();
}

void initializeWeights() {
  Serial.println("Initializing final layer weights...");

  for (int i = 0; i < 16; i++) {
    for (int j = 0; j < 4; j++) {
      final_layer_weights[i][j] = random(-100, 100) / 1000.0;  // -0.1 to 0.1
    }
  }

  for (int i = 0; i < 4; i++) {
    final_layer_biases[i] = 0.0;
  }
}

void collectTrainingSamples(int count) {
  Serial.print("Collecting ");
  Serial.print(count);
  Serial.println(" training samples...");

  for (int i = 0; i < count && num_samples < MAX_SAMPLES; i++) {
    int label = random(0, 4);  // Random label 0-3

    // Generate characteristic pattern for this label
    for (int j = 0; j < INPUT_SIZE; j++) {
      float value = 0.0;

      switch(label) {
        case 0:  // Pattern 1
          value = sin(j * 0.1) + random(-20, 20) / 100.0;
          break;
        case 1:  // Pattern 2
          value = cos(j * 0.1) + random(-20, 20) / 100.0;
          break;
        case 2:  // Pattern 3
          value = sin(j * 0.2) * cos(j * 0.1) + random(-20, 20) / 100.0;
          break;
        case 3:  // Pattern 4
          value = random(-50, 50) / 100.0;
          break;
      }

      training_buffer[num_samples].input[j] = value;
    }

    training_buffer[num_samples].label = label;
    num_samples++;
  }

  Serial.print("Total samples collected: ");
  Serial.println(num_samples);
}

void trainModel(int epochs) {
  Serial.println("\n========================================");
  Serial.println("TRAINING");
  Serial.println("========================================\n");

  for (int epoch = 0; epoch < epochs; epoch++) {
    float total_loss = 0.0;
    int batches = 0;

    // Shuffle samples
    for (int i = num_samples - 1; i > 0; i--) {
      int j = random(0, i + 1);
      TrainingSample temp = training_buffer[i];
      training_buffer[i] = training_buffer[j];
      training_buffer[j] = temp;
    }

    // Mini-batch training
    for (int batch_start = 0; batch_start < num_samples; batch_start += batch_size) {
      int batch_end = min(batch_start + batch_size, num_samples);

      // Accumulate gradients over batch
      float weight_gradients[16][4] = {0};
      float bias_gradients[4] = {0};

      for (int i = batch_start; i < batch_end; i++) {
        // Forward pass through feature extractor
        float features[16];
        extractFeatures(training_buffer[i].input, features);

        // Forward pass through final layer
        float logits[4];
        for (int j = 0; j < 4; j++) {
          logits[j] = final_layer_biases[j];
          for (int k = 0; k < 16; k++) {
            logits[j] += features[k] * final_layer_weights[k][j];
          }
        }

        // Softmax
        float max_logit = logits[0];
        for (int j = 1; j < 4; j++) {
          if (logits[j] > max_logit) max_logit = logits[j];
        }

        float sum_exp = 0.0;
        float probs[4];
        for (int j = 0; j < 4; j++) {
          probs[j] = exp(logits[j] - max_logit);
          sum_exp += probs[j];
        }
        for (int j = 0; j < 4; j++) {
          probs[j] /= sum_exp;
        }

        // Cross-entropy loss
        int true_label = training_buffer[i].label;
        total_loss += -log(probs[true_label] + 1e-7);

        // Backward pass - compute gradients
        for (int j = 0; j < 4; j++) {
          float error = probs[j] - (j == true_label ? 1.0 : 0.0);

          bias_gradients[j] += error;

          for (int k = 0; k < 16; k++) {
            weight_gradients[k][j] += error * features[k];
          }
        }
      }

      // Update weights
      int batch_count = batch_end - batch_start;
      for (int i = 0; i < 16; i++) {
        for (int j = 0; j < 4; j++) {
          final_layer_weights[i][j] -= learning_rate * weight_gradients[i][j] / batch_count;
        }
      }

      for (int i = 0; i < 4; i++) {
        final_layer_biases[i] -= learning_rate * bias_gradients[i] / batch_count;
      }

      batches++;
    }

    float avg_loss = total_loss / num_samples;

    if ((epoch + 1) % 5 == 0) {
      Serial.print("Epoch ");
      Serial.print(epoch + 1);
      Serial.print("/");
      Serial.print(epochs);
      Serial.print(" - Loss: ");
      Serial.print(avg_loss, 4);
      Serial.print(" - LR: ");
      Serial.println(learning_rate, 4);
    }

    // Decay learning rate
    learning_rate *= lr_decay;
  }

  Serial.println("\nTraining complete!");
}

void extractFeatures(float* input, float* features) {
  // Run inference through base model to get features
  // (In real implementation, we'd extract intermediate layer output)
  // For simplicity, we'll simulate feature extraction

  TfLiteTensor* model_input = interpreter->input(0);
  for (int i = 0; i < INPUT_SIZE; i++) {
    model_input->data.f[i] = input[i];
  }

  interpreter->Invoke();

  // Simulate extracting 16-dimensional features
  // (Would come from second-to-last layer in real implementation)
  for (int i = 0; i < 16; i++) {
    features[i] = sin(input[i * 9] + input[i * 9 + 1]);
  }
}

float evaluateModel() {
  // Generate test set
  int test_size = 40;
  int correct = 0;

  for (int i = 0; i < test_size; i++) {
    int true_label = random(0, 4);
    float test_input[INPUT_SIZE];

    // Generate test pattern
    for (int j = 0; j < INPUT_SIZE; j++) {
      float value = 0.0;
      switch(true_label) {
        case 0: value = sin(j * 0.1) + random(-20, 20) / 100.0; break;
        case 1: value = cos(j * 0.1) + random(-20, 20) / 100.0; break;
        case 2: value = sin(j * 0.2) * cos(j * 0.1) + random(-20, 20) / 100.0; break;
        case 3: value = random(-50, 50) / 100.0; break;
      }
      test_input[j] = value;
    }

    // Predict
    float features[16];
    extractFeatures(test_input, features);

    float logits[4];
    for (int j = 0; j < 4; j++) {
      logits[j] = final_layer_biases[j];
      for (int k = 0; k < 16; k++) {
        logits[j] += features[k] * final_layer_weights[k][j];
      }
    }

    int predicted_label = 0;
    float max_logit = logits[0];
    for (int j = 1; j < 4; j++) {
      if (logits[j] > max_logit) {
        max_logit = logits[j];
        predicted_label = j;
      }
    }

    if (predicted_label == true_label) {
      correct++;
    }
  }

  return (float)correct / test_size;
}

void saveWeights() {
  Serial.println("\nSaving updated weights to flash...");
  preferences.begin("ml_weights", false);

  // Save weights as byte array
  preferences.putBytes("weights", final_layer_weights, sizeof(final_layer_weights));
  preferences.putBytes("biases", final_layer_biases, sizeof(final_layer_biases));

  preferences.end();
  Serial.println("Weights saved!");
}

void loadTrainingData() {
  preferences.begin("ml_data", true);
  num_samples = preferences.getInt("count", 0);
  preferences.end();

  Serial.print("Loaded ");
  Serial.print(num_samples);
  Serial.println(" previous samples");
}

void loop() {
  // Nothing in loop
}
```

**Expected Outcomes:**
- Initial accuracy: 25-35% (random-ish)
- After 20 epochs: 70-85% (significant improvement)
- Training time: ~10-30 seconds
- Weights successfully persisted to flash
- Demonstrates on-device adaptation capability

**Key Insights:**
- On-device learning is feasible for simple tasks
- Final layer fine-tuning is most practical
- Flash memory enables persistent learning
- Trade-off: training time vs. adaptation benefit
:::

---

## Summary

These Level 3 exercises covered:

1. **End-to-end TFLite deployment**: Training, conversion, quantization, and hardware deployment
2. **Memory optimization**: Finding minimum tensor arena through systematic testing
3. **Performance benchmarking**: Statistical profiling of inference latency and throughput
4. **Multi-model systems**: Dynamic model switching with graceful degradation
5. **On-device learning**: Implementing simple gradient descent for model adaptation

**Key Takeaways:**
- Memory is the primary constraint on microcontrollers
- INT8 quantization provides 4× compression with minimal accuracy loss
- Performance varies significantly across hardware platforms
- Multiple models enable adaptive behavior based on constraints
- Simple on-device learning is practical for specific use cases
