Edge AI - Convert training speech data to micro feature.

In micro speech project we can use training data and convert it to micro feature so that we can test model accuracy in MCU without bothering MIC gain or signal chain issue.

Tensorflow provide a python script to generate feature from a input data.

Steps:

1. Download datasets.

2. Using tensorflow/tensorflow/examples/speech_commands/wav_to_features.py to convert input data. (!!Notice: Must use --preprocess=micro for tensorflow micro model comparability.

Here is my colab note book to generate 10 random feature with specific keyword.

Ref.

 1. https://stackoverflow.com/questions/701402/best-way-to-choose-a-random-file-from-a-directory

 2.https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/examples/micro_speech/train/README.md

Edge AI - Train micro_speech model and deploy to NXP FRDM-K66F.

Tensorflow lite micro_speech example provide training method for speech recognize model to different key word detect.

In this article we use Google Colaboratory to train micro_speech model.

1. Open note book and change WANTED_WORDS
    I choose ON/OFF for my project.

# A comma-delimited list of the words you want to train for.
# The options are: yes,no,up,down,left,right,on,off,stop,go
# All the other words will be used to train an "unknown" label and silent
# audio data with no spoken words will be used to train a "silence" label.
WANTED_WORDS = "on,off"

2. Follow instruction and you can see training  progress(Model accuracy for training set and validation set) in tensorboard section.

3. Convert to TF-lite model and test TensorFlow Lite model's accuracy

5. Modify micro_speech/micro_features/micro_model_settings.cc kCategoryLabels to on/off

6. Replace micro_speech/micro_features/model.cc  to new trained model.

7. Build and flash 

8. Test result 

I use K66F onboard mic to test and the accuracy is lower than I expect. But the trained model has 80% more accuracy, so maybe need to use line mic to get better accuracy speech recognize result.

Edge AI - Modify micro speech example MIC gain on NXP FRDM-K66F for better accuracy.

 The example code recognize accuracy is lower than I expect. So I try to increase microphone gain from 0 dB to +6 dB and the result is better now.

1. Change DA7212 configuration in tensorflow/lite/micro/examples/micro_speech/nxp_k66f/audio_provider.cc

volatile uint8_t g_da7212_register_config[da7212ConfigurationSize][2] = {

...

   {0x34, 0x07},                              // Change MIXIN to 6 dB

    {0x35, 0x07},                         // Change MIXIN to 6 dB

....

2. Build and flash new binary to NXP_K66F board and test.

Ref 

DA7212 datasheet 

Edge AI - Run tensorflow lite micro speech expample on NXP FRDM-K66F.

 Step to build tf-lite micro speach example for nxp_k66f

1. Clone tensorflow code

  $>git clone https://github.com/tensorflow/tensorflow.git

  

2. Install mbed-cli 

     $>sudo apt install python3 python3-pip git mercurial

     $>python3 -m pip install mbed-cli

3. Install ARM cross compile toolchain.(In this project I use GNU Arm Embedded Toolchain)

https://developer.arm.com/tools-and-software/open-source-software/developer-tools/gnu-toolchain/gnu-rm/downloads

4. Configure mbed cli

  $>mbed config -G  GCC_ARM_PATH=/YOUR_COMPILER_INSTALL_PATH/gcc-arm-none-eabi-9-2020-q2-update/bin

5. Generate mbed project for nxp_k66f

  $>cd tensorflow

  $>make -f tensorflow/lite/micro/tools/make/Makefile TARGET=mbed TAGS="nxp_k66f" generate_micro_speech_mbed_project

6.  Create mbed project 

  $>cd tensorflow/lite/micro/tools/make/gen/mbed_cortex-m4/prj/micro_speech/mbed

  $>mbed new  .

7. Configure porject usin C++ 11

python -c 'import fileinput, glob;

for filename in glob.glob("mbed-os/tools/profiles/*.json"):

  for line in fileinput.input(filename, inplace=True):

    print line.replace("\"-std=gnu++14\"","\"-std=c++11\", \"-fpermissive\"")'

 

8 Connect the USB cable to nxp_k66f

9. Compile and flash project, after flash complete reset the board.

mbed compile --target K66F --toolchain GCC_ARM --profile release --flash

10. Connect to nxp_k66f serial port with baudrate 14400.

11. Saying Yes and NO and will print repsonse word in serial port, since nxp_k66f has onboard microphone you can direct speak to it and see the result in terminal.

12. Test result 

I want using this sample code to build a IOT switch that can control with speach, So  in next article I will try to retrain model with 'ON' and 'OFF' key word and try to deploy to nxp_k66f

Ref

Mbed

https://os.mbed.com/docs/mbed-os/v6.9/build-tools/install-and-set-up.html

Tensorflow lite

https://github.com/tensorflow/tensorflow/tree/114b8ef31ac66155ec9b0590bc7115125f7fe61e/tensorflow/lite/micro/examples/micro_speech#deploy-to-nxp-frdm-k66f

NXP_K66F user guide

https://www.nxp.com/docs/en/user-guide/FRDMK66FUG.pdf

Machine Learning Foundations NLP(2): Using the Sequencing APIs

Using the Sequencing APIs

Sequencing is use to format sentence array using token,

For example:

"I have a car"

"Car have a door"

 

Tokenize: [I:1] [have:2] [a:3] [car:4] [door:5]

 

then these two sentence can be represent to:

[1 2 3 4] 

[4 2 3 5]

Sequencing is useful to represent sentence data and take as input for a neuron network.

Code

Result:

Reference:

Machine Learning Foundations: Ep #9 - Using the Sequencing APIs

Machine Learning Foundations NLP(1): Tokenization for Natural Language Processing

Tokenization for Natural Language Processing

Tokenize is mean to break down a sentence to server work, for example:

I have a car. -> I / have / a/ car

Tensorflow provide a tokenize tool:Tokenizer, it can easily to use for tokenize input sentence.

Code:

Result:

{'have': 1, 'a': 2, 'i': 3, 'he': 4, 'apple': 5, 'bike': 6, 'pen': 7, 'car': 8}

Reference:

Machine Learning Foundations: Ep #8 - Tokenization for Natural Language Processing

DeepSpeech: Speech to text AI model.

• Overview

"DeepSpeech is an open source Speech-To-Text engine, using a model trained by machine learning techniques based on Baidu’s Deep Speech research paper."

DeepSpeech provide lots of lauange api support, Python Javascript, c, and it's easily use to involve in application

• Install DeepSpeech

Follow user guide instruction.

• Demo

Using command line tool to inference sound data.

$> deepspeech --model deepspeech-0.7.0-models.pbmm --audio audio/2830-3980-0043.wav

Output:

Loading model from file deepspeech-0.7.0-models.pbmm
TensorFlow: v1.15.0-24-gceb46aa
DeepSpeech: v0.7.1-0-g2e9c281
Loaded model in 0.0093s.
Loading scorer from files deepspeech-0.7.0-models.scorer
Loaded scorer in 0.00023s.
Running inference.
experience proves this
Inference took 1.480s for 1.975s audio file.

 

The red color string is inference text data of input sound data.

Using Python API to inference sound data.

Output:

your power is sufficient i said sound data

Reference:

1. DeepSpeech 0.7.x Examples
2. DeepSpeech Document
3. Python library-wave

Machine Learning Foundations: Exercise 4 Happy and sad image classify model with 99.9% accuracy .

Machine Learning Foundations: Exercise 4 Happy and sad image classify model with 99.9% accuracy:code lab link

Build a mode to classify happy and sad model with convolution neuron network. with more 99.9% accuracy.

 

Note:

•  Model over-fitting: Near 100% accuracy on training data, but lower accuracy on testing data.
•  ImageDataGenerator:  Label data automatic using directory name.

Code: 

Machine Learning Foundations: Exercise 3 Improve accuracy of MNIST using Convolutions.

Exercise 3: Improve accuracy of MNIST using Convolutions:code lab link

Improve MNIST to 99.8% accuracy or more using only a single convolutional layer and a single MaxPooling 2D.

The filter amount will affect the accuracy and training time.

Code: 

Machine Learning Foundations: Exercise 2 Handwriting digit model with 99% accuracy

Exercise 2: Handwriting digit model with 99% accuracy:code lab link

Write an MNIST classifier that trains to 99% accuracy or above, and does it without a fixed number of epochs -- i.e. you should stop training once you reach that level of accuracy.

Code:

Result:

Machine Learning Foundations : Exercise 1 House Price Question

Exercise 1 : House Prices Question : code lab link

Build a neural network that predicts the price of a house according to a simple formula, house pricing was as easy as a house costs 50k + 50k per bedroom, so that a 1 bedroom house costs 100k, a 2 bedroom house costs 150k etc.

Training data set

Bedroom amount [1, 2, 3, 4,]
House price [100, 150, 200, 250]

Code: