Introduction: Vegetables and Fruits Ripeness Detection by Color W/ TensorFlow

By GitHub
About: AI & Full-Stack Developer | @EdgeImpulse | @Particle | Maker | Independent Researcher

Collate spectral color data of varying fruits and vegetables and interpret this data set with a neural network to predict ripening stages.

Supplies

1 X Arduino Nano 33 IoT

1 X DFRobot AS7341 11-Channel Visible Light Sensor

1 X Raspberry Pi 3B+ or 4

1 X 10K Potentiometer (Long-Shaft)

1 X Potentiometer Knob

4 X Button (6x6)

1 X 5 mm LED: Green

1 X 5 mm LED: Red

1 X Breadboard

Step 1: Story

Thanks to our evolved acuity, it is not struggling for us to detect ripeness by the appearance of a fruit or vegetable. However, nowadays, foods are a subject of mass production. Thus, we need a more reliable and effortless method to distinguish fruits and vegetables depending on their ripeness than the naked eye. In the hope of assisting food companies, grocery stores, and farmers in sorting fruits and vegetables, I developed this device that detects ripening stages by spectral color with a neural network model.

As fruits and vegetables ripen, they change color because of the four families of pigments:

  • chlorophyll (green)
  • carotenoids (yellow, red, orange)
  • flavonoids: anthocyanins + anthoxanthins (red, blue, purple)
  • betalains (red, yellow, purple)

These pigments are groups of molecular structures absorbing a specific set of wavelengths and reflecting the remainder. Unripe fruits are green because of chlorophyll in their cells. As they ripen, the chlorophyll breaks down and is replaced by orange carotenoids and red anthocyanins. These compounds are antioxidants that prevent the fruit from spoiling too quickly in the air. Then, the enzymatic browning occurs and causes discoloration - turning brown. Enzymes function as a catalyst for chemical reactions instigating discoloration, such as:

  • hydroxylation of phenols
  • oxidation of phenols

After doing some research on color-changing processes as fruits and vegetables ripen, I decided to build an artificial neural network (ANN) based on the classification model to interpret the spectral color of varying fruits and vegetables to predict ripening stages.

Before building and testing my neural network model, I developed a web application in PHP to collate the spectral color data of fruits and vegetables generated by the AS7341 visible light sensor to create a ripening stages data set by spectral color. I used an Arduino Nano 33 IoT to send the data produced by the visible light sensor to the web application. Empirically, I assigned a ripening stage (label) while obtaining spectral color data for each fruit and vegetable by using four class buttons connected to the Nano 33 IoT:

  • Early Ripe
  • Partially Ripe
  • Ripe
  • Decay

After completing the data set, I built my artificial neural network (ANN) with TensorFlow to make predictions on the ripening stages (labels) based on spectral color.

Then, after testing, I conducted experiments with my neural network model to predict the ripening stages of varying fruits and vegetables by spectral color. As far as my experiments go, the model is working impeccably.

So, this is my project in a nutshell 😃

In the following steps, you can find more detailed information on coding, data collection, and building an artificial neural network model with TensorFlow.

Huge thanks to DFRobot for sponsoring this project.

Sponsored products by DFRobot:

⭐ DFRobot AS7341 11-Channel Visible Light Sensor | Inspect

Step 2: Developing a Web Application in PHP to Collate Spectral Color Data of Fruits and Vegetables

I developed this web application (Vegetables and Fruits Data Logger) in PHP to obtain spectral color data of fruits and vegetables from the AS7341 visible light sensor via the Arduino Nano 33 IoT and insert this data into a CSV file (spectral_color_database.csv) to create a ripening stages data set by spectral color.

The web application includes one file (index.php) and requires these parameters to insert a new row (entry) to the data set:

  • F1
  • F2
  • F3
  • F4
  • F5
  • F6
  • F7
  • F8
  • nir_1
  • nir_2
  • class

💻 index.php

⭐ Check whether all required parameters are transferred by the Nano 33 IoT.

⭐ If so, create the data array with the obtained parameters, including the current date.

⭐ Insert the recently generated data array into the spectral_color_database.csv file as a new row by using the fputcsv function.

⭐ Then, print: Data Inserted Successfully!

⭐ If there are omissions or no incoming data, print: Waiting for data from the AS7341 sensor to insert...

# Check the incoming data packet:
if(isset($_GET["F1"]) && isset($_GET["F2"]) && isset($_GET["F3"]) && isset($_GET["F4"]) && isset($_GET["F5"]) && isset($_GET["F6"]) && isset($_GET["F7"]) && isset($_GET["F8"]) && isset($_GET["nir_1"]) && isset($_GET["nir_2"]) && isset($_GET["class"])){
	# Create the data array.
	$data = array($_GET["F1"], $_GET["F2"], $_GET["F3"], $_GET["F4"], $_GET["F5"], $_GET["F6"], $_GET["F7"], $_GET["F8"], $_GET["nir_1"], $_GET["nir_2"], $_GET["class"], date("m/d"));
    # Insert the recently generated data array into the CSV file as a new row.	
	$file = fopen("spectral_color_database.csv", "a");
	fputcsv($file, $data);
	fclose($file);
	// Print result:
	echo "Data Inserted Successfully!";
}else{
	echo "Waiting for data from the AS7341 sensor to insert...";
}

Step 3: Setting Up the Web Application on Raspberry Pi

After creating the PHP web application, I decided to run it on my Raspberry Pi, but you can run the application on any server as long as it is a PHP server.

If you want to use Raspberry Pi but do not know how to set up a LAMP web server on it, you can inspect this tutorial.

⭐ First of all, extract the Vegetables_and_Fruits_Data_Logger.zip folder.

⭐ Then, move the application folder (Vegetables_and_Fruits_Data_Logger) to the apache server (/var/www/html) by using the terminal since the apache server is a protected location.

sudo mv /home/pi/Downloads/Vegetables_and_Fruits_Data_Logger/ /var/www/html/

⭐ If the required parameters are not set nor found, the web application prints: Waiting for data from the AS7341 sensor to insert...

http://localhost/Vegetables_and_Fruits_Data_Logger/

⭐ Otherwise, the web application inserts the incoming data as a new row to the CSV file (data set) by adding the current date and prints: Data Inserted Successfully!

http://localhost/Vegetables_and_Fruits_Data_Logger/?F1=13&F2=12&F3=12&F4=12&F5=12&F6=12&F7=12&F8=12&nir_1=12&nir_2=9&class=0

Step 4: Setting Up the Arduino Nano 33 IoT

If you are a novice in programming with the Arduino Nano 33 IoT, follow the steps below to program it with the Arduino IDE.

⭐ Download the required drivers - Arduino SAMD Core - as explained here.

⭐ To be able to connect to the WiFi, download the WiFiNINA library from here.

Step 5: Programming the Arduino Nano 33 IoT to Obtain the Required Parameters and Send Them to the Web Application

First of all, to be able to obtain spectral color data from the AS7341 visible light sensor with the Arduino Nano 33 IoT, you need to install the DFRobot AS7341 library.

⭐ Include required libraries and define the WiFi settings.

char ssid[] = "SSID";        // your network SSID (name)
char pass[] = "PASSWORD";    // your network password (use for WPA, or use as key for WEP)
int keyIndex = 0;            // your network key Index number (needed only for WEP)
int status = WL_IDLE_STATUS;

// Enter the IPAddress of your Raspberry Pi.
IPAddress server(192, 168, 1, 20);

// Initialize the Ethernet client library
WiFiClient client; /* WiFiSSLClient client; */

⭐ Define the AS7341 visible light sensor settings and objects.

// Define the AS7341 object.
DFRobot_AS7341 as7341;
// Define AS7341 data objects:
DFRobot_AS7341::sModeOneData_t data1;
DFRobot_AS7341::sModeTwoData_t data2;

⭐ Detect if I2C can communicate properly and enable the built-in LED on the AS7341 sensor.

  // Detect if I2C can communicate properly 
  while (as7341.begin() != 0) {
    Serial.println("I2C init failed, please check if the wire connection is correct");
    delay(1000);
  }

  // Enable the built-in LED on the AS7341 sensor.
  as7341.enableLed(true);

⭐ Check for the WiFi module to verify the connection.

  if (WiFi.status() == WL_NO_MODULE) { Serial.println("Connection Failed!"); while (true); }
  // Attempt to connect to the WiFi network:
  while (status != WL_CONNECTED) {
    Serial.println("Attempting to connect to WiFi !!!");
    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);
    // Wait 10 seconds for connection:
    delay(10000);
  }

⭐ In the read_controls function, read potentiometer and four class button values.

void read_controls(){
  // Potentiometer:
  pot_val = analogRead(pot);
  // Class Buttons:
  class_1_val = digitalRead(class_1);
  class_2_val = digitalRead(class_2);
  class_3_val = digitalRead(class_3);
  class_4_val = digitalRead(class_4);
}

⭐ In the adjust_brightness function, set pin current to control the brightness (1~20 corresponds to current 4mA, 6mA, 8mA, 10mA, 12mA, ......, 42mA) of the built-in LED on the AS7341 sensor.

⭐ Start spectrum measurement with the AS7341 sensor and read the value of sensor data channel 0~5 under these channel mapping modes:

  • eF1F4ClearNIR
  • eF5F8ClearNIR

⭐ Then, print spectral color data generated by the sensor:

  // Start spectrum measurement:
  // Channel mapping mode: 1.eF1F4ClearNIR
  as7341.startMeasure(as7341.eF1F4ClearNIR);
  // Read the value of sensor data channel 0~5, under eF1F4ClearNIR
  data1 = as7341.readSpectralDataOne();
  // Channel mapping mode: 2.eF5F8ClearNIR
  as7341.startMeasure(as7341.eF5F8ClearNIR);
  // Read the value of sensor data channel 0~5, under eF5F8ClearNIR
  data2 = as7341.readSpectralDataTwo();
  // Print data:
  Serial.print("F1(405-425nm): "); Serial.println(data1.ADF1);
  Serial.print("F2(435-455nm): "); Serial.println(data1.ADF2);
  Serial.print("F3(470-490nm): "); Serial.println(data1.ADF3);
  Serial.print("F4(505-525nm): "); Serial.println(data1.ADF4);
  Serial.print("F5(545-565nm): "); Serial.println(data2.ADF5);
  Serial.print("F6(580-600nm): "); Serial.println(data2.ADF6);
  Serial.print("F7(620-640nm): "); Serial.println(data2.ADF7);
  Serial.print("F8(670-690nm): "); Serial.println(data2.ADF8);
  // CLEAR and NIR:
  Serial.print("Clear_1: "); Serial.println(data1.ADCLEAR);
  Serial.print("NIR_1: "); Serial.println(data1.ADNIR);
  Serial.print("Clear_2: "); Serial.println(data2.ADCLEAR);
  Serial.print("NIR_2: "); Serial.println(data2.ADNIR);
  Serial.print("\n------------------------------\n");
  delay(1000);

⭐ Send spectral color data to the PHP web application by assigning a ripening stage label [0 - 3] depending on the pressed class button:

  • 0 — Early Ripe
  • 1 — Partially Ripe
  • 2 — Ripe
  • 3 — Decay
  if(!class_1_val) make_a_get_request("/Vegetables_and_Fruits_Data_Logger/", "0");
  if(!class_2_val) make_a_get_request("/Vegetables_and_Fruits_Data_Logger/", "1");
  if(!class_3_val) make_a_get_request("/Vegetables_and_Fruits_Data_Logger/", "2");
  if(!class_4_val) make_a_get_request("/Vegetables_and_Fruits_Data_Logger/", "3");

⭐ In the make_a_get_request function:

⭐ Connect to the web application named Vegetables and Fruits Data Logger.

⭐ If the connection is successful, create the query string with the spectral color data and the selected ripening stage label. Otherwise, turn on the 5mm red LED.

⭐ Then, make a Get request with the query string to send the required parameters to the web application.

⭐ If there are incoming bytes available, get the response from the web application.

⭐ If the transferred data is inserted into the data set (spectral_color_database.csv) successfully by the web application, turn on the 5mm green LED:

response.indexOf("Data Inserted Successfully!") > 0

⭐ Otherwise, turn on the 5mm red LED.

void make_a_get_request(String application, String _class){
  // Connect to the web application named Vegetables_and_Fruits_Data_Logger. Change '80' with '443' if you are using SSL connection.
  if(client.connect(server, 80)){
    // If successful:
    Serial.println("\n\nConnected to the server!");
    // Create the query string:
    String query = application + "?F1="+data1.ADF1+"&F2="+data1.ADF2+"&F3="+data1.ADF3+"&F4="+data1.ADF4+"&F5="+data2.ADF5+"&F6="+data2.ADF6+"&F7="+data2.ADF7+"&F8="+data2.ADF8+"&nir_1="+data1.ADNIR+"&nir_2="+data2.ADNIR+"&class="+_class;
    // Make an HTTP Get request:
    client.println("GET " + query + " HTTP/1.1");
    client.println("Host: 192.168.1.20");
    client.println("Connection: close");
    client.println();
  }else{
    Serial.println("Server Error!");
    digitalWrite(red, HIGH);
  }
  delay(2000); // Wait 2 seconds after connection...
  // If there are incoming bytes available, get the response from the web application.
  String response = "";
  while (client.available()) { char c = client.read(); response += c; }
  if(response != ""){
    Serial.println(response);
    Serial.println("\n");
    // Check whether the transferred data is inserted successfully or not:
    if(response.indexOf("Data Inserted Successfully!") > 0){
      digitalWrite(green, HIGH);
    }else{
      digitalWrite(red, HIGH);
    }
  }
  // Turn off LEDs:
  delay(3000);
  digitalWrite(green, LOW);
  digitalWrite(red, LOW);
}

⭐ By serial communication, while running, the Arduino Nano 33 IoT prints:

  • the brightness level of the built-in LED on the AS7341 sensor
  • spectral color data generated by the AS7341 sensor
  • server response from the web application

Step 6: Creating the Ripening Stages Data Set Based on Spectral Color

To train my neural network model on ripening stages by spectral color, I needed to collect spectral color data of varying fruits and vegetables to create a data set with eminent validity. Hence, I decided to use the AS7341 visible light sensor to collect spectral color data.

This sensor employs the new generation of AS7341 spectral sensor IC launched by the well-known AMS company. It features eight channels for visible light, one channel for near-IR (NIR), one channel without a filter (Clear), and an extra channel distributed over ambient light flicker (Flicker). Also, it has six parallel 16-bit ADC channels for signal and data processing. Thanks to the two integrated LEDs with adjustable brightness, it can collect data even in a dark environment.

#️⃣ While running, I collected these parameters with the AS7341 visible light sensor:

  • F1 (405-425nm)
  • F2 (435-455nm)
  • F3 (470-490nm)
  • F4 (505-525nm)
  • F5 (545-565nm)
  • F6 (580-600nm)
  • F7 (620-640nm)
  • F8 (670-690nm)
  • Clear_1
  • NIR_1
  • Clear_2
  • NIR_2

#️⃣ As explained in the previous step, I connected the AS7341 sensor to the Arduino Nano 33 IoT to send parameters and the assigned ripening stage labels to the PHP web application. After obtaining data, the web application inserts the required parameters by adding the current date parameter (Step 1) into the spectral_color_database.csv file (data set) under these columns:

  • F1
  • F2
  • F3
  • F4
  • F5
  • F6
  • F7
  • F8
  • NIR_1
  • NIR_2
  • Ripeness
  • Date

#️⃣ After completing programming and assembling the device, I selected eleven different fruits and vegetables to collect their ripeness classes by spectral color:

#️⃣ The device allows the user to send data to the web application and assign a ripeness class (label) by pressing one of the four class buttons:

  • 0 — Early Ripe
  • 1 — Partially Ripe
  • 2 — Ripe
  • 3 — Decay

#️⃣ For each fruit and vegetable, I collated spectral color data and the empirically assigned ripeness class for ten days, three times a day:

#️⃣ The device lets the user adjust the brightness of the built-in LED on the AS7341 sensor with the potentiometer.

#️⃣ If the transferred data is inserted into the data set successfully, the device turns on the 5mm green LED. Otherwise, it turns on the 5mm red LED.

As expected, fruits and vegetables had gradually gone stale during ten days of data collecting:

Finally, after collecting ripeness classes by spectral color three times a day for ten days, I elicited a data set (spectral_color_database.csv) with eminent validity to train my neural network model.

Step 7: Building an Artificial Neural Network (ANN) With TensorFlow

When I completed my ripening stages by spectral color data set, I started to work on my artificial neural network (ANN) model to make predictions on the ripening stages of varying fruits and vegetables.

I decided to create my neural network model with TensorFlow in Python. Thus, first of all, I followed the steps below to grasp a better understanding of my data set:

  • Data Visualization
  • Data Scaling (Normalizing)
  • Data Preprocessing
  • Data Splitting

As explained in the previous steps, I empirically assigned ripening stage classes for each fruit and vegetable entry (input) while collecting spectral color data. Hence, I used these pre-defined classes as labels and did not need to preprocess my data set for assigning labels:

  • 0 — Early Ripe
  • 1 — Partially Ripe
  • 2 — Ripe
  • 3 — Decay

After scaling (normalizing) and preprocessing inputs in my data set, I elicited ten input variables and one label for each fruit and vegetable entry, classified with the four mentioned classes. Then, I built an artificial neural network model with TensorFlow and trained it with my data set to obtain the best possible results and predictions.

Layers:

  • 10 [Input]
  • 16 [Hidden]
  • 32 [Hidden]
  • 64 [Hidden]
  • 128 [Hidden]
  • 256 [Hidden]
  • 4 [Output]

To execute all steps above, I created a class named Ripeness_Detection in Python after including the required libraries:

import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

Subsequently, I will discuss coding in Python for each step I mentioned above.

Also, you can download build_neural_network_model.py to inspect coding.

Step 8: Visualizing the Ripening Stages Data Set Based on Spectral Color

Before diving into building a neural network model, it is crucial to understand the given data set to pass the model appropriately formatted data.

So, in this step, I will show you how to visualize the ripening stages data set and scale (normalize) it in Python.

⭐ First of all, read the ripening stages data set from spectral_color_database.csv.

csv_path = "E:\PYTHON\Vegetables_and_Fruits_Ripeness_Detection\spectral_color_database.csv"
df = pd.read_csv(csv_path)

⭐ In the graphics function, visualize the requested columns from the ripening stages data set by using the Matplotlib library.

    def graphics(self, column_1, column_2, x_label, y_label):
        # Show requested columns from the data set:
        plt.style.use("dark_background")
        plt.gcf().canvas.set_window_title('Vegetables and Fruits Ripeness Detection by Color')
        plt.hist2d(self.df[column_1], self.df[column_2], cmap='RdBu')
        plt.colorbar()
        plt.xlabel(x_label)
        plt.ylabel(y_label)
        plt.title(x_label)
        plt.show()

⭐ In the data_visualization function, scrutinize all columns before scaling and preprocessing ripening stages data set to build a model with appropriately formatted data.

    def data_visualization(self):
        # Scrutinize requested columns to build a model with appropriately formatted data:
        self.graphics('F1', 'NIR_1', 'F1(405-425nm)', 'Near-Infrared')
        self.graphics('F2', 'NIR_1', 'F2(435-455nm)', 'Near-Infrared')
        self.graphics('F3', 'NIR_1', 'F3(470-490nm)', 'Near-Infrared')
        self.graphics('F4', 'NIR_1', 'F4(505-525nm)', 'Near-Infrared')
        self.graphics('F5', 'NIR_2', 'F5(545-565nm)', 'Near-Infrared')
        self.graphics('F6', 'NIR_2', 'F6(580-600nm)', 'Near-Infrared')
        self.graphics('F7', 'NIR_2', 'F7(620-640nm)', 'Near-Infrared')
        self.graphics('F8', 'NIR_2', 'F8(670-690nm)', 'Near-Infrared')

Step 9: Assigning Labels (pre-defined Ripening Stages) and Scaling (normalizing) the Input Data

Before building and training my neural network model, I utilized pre-defined ripeness classes for each food and vegetable entry (input) as its label.

⭐ In the define_and_assign_labels function, get pre-defined ripeness classes (labels) [0 - 3] for each input in the data set and append them to the labels array.

    def define_and_assign_labels(self):
        self.labels = self.df.pop("Ripeness")

After assigning labels, I needed to create inputs to train my neural network model. From my ripening stages data set, I decided to use these ten parameters, generated by the AS7341 visible light sensor as spectral color data, to create inputs:

  • F1
  • F2
  • F3
  • F4
  • F5
  • F6
  • F7
  • F8
  • NIR_1
  • NIR_2

After selecting ten parameters for inputs and visualizing the data set, I scaled (normalized) each parameter column to format them properly.

Normally, each input for a fruit or vegetable in the data set looked like this before scaling:

  • 131, 148, 184, 765, 1000, 1000, 1000, 809, 276, 273

After completing scaling (normalizing), I extracted these scaled parameter columns from the ripening stages data set for each fruit and vegetable entry (input):

  • scaled_F1
  • scaled_F2
  • scaled_F3
  • scaled_F4
  • scaled_F5
  • scaled_F6
  • scaled_F7
  • scaled_F8
  • scaled_NIR_1
  • scaled_NIR_2

Input with the scaled parameters:

  • [0.131 0.148 0.184 0.765 1 1 1 0.809 0.276 0.273]

⭐ In the scale_data_and_define_inputs function, divide every parameter column into their required values in order to make them smaller than or equal to 1.

⭐ Then, create inputs with the scaled parameters, append them to the inputs array, and convert this array to a NumPy array by using the asarray() function.

⭐ Each input includes ten parameters [shape=(10,)]:

  • [scaled_F1, scaled_F2, scaled_F3, scaled_F4, scaled_F5, scaled_F6, scaled_F7, scaled_F8, scaled_NIR_1, scaled_NIR_2]
    def scale_data_and_define_inputs(self):
        self.df["scaled_F1"] = self.df.pop("F1") / 1000
        self.df["scaled_F2"] = self.df.pop("F2") / 1000
        self.df["scaled_F3"] = self.df.pop("F3") / 1000
        self.df["scaled_F4"] = self.df.pop("F4") / 1000
        self.df["scaled_F5"] = self.df.pop("F5") / 1000
        self.df["scaled_F6"] = self.df.pop("F6") / 1000
        self.df["scaled_F7"] = self.df.pop("F7") / 1000
        self.df["scaled_F8"] = self.df.pop("F8") / 1000
        self.df["scaled_NIR_1"] = self.df.pop("NIR_1") / 1000
        self.df["scaled_NIR_2"] = self.df.pop("NIR_2") / 1000
        # Create the inputs array using the scaled variables:
        for i in range(len(self.df)):
            self.inputs.append(np.array([self.df["scaled_F1"][i], self.df["scaled_F2"][i], self.df["scaled_F3"][i], self.df["scaled_F4"][i], self.df["scaled_F5"][i], self.df["scaled_F6"][i], self.df["scaled_F7"][i], self.df["scaled_F8"][i], self.df["scaled_NIR_1"][i], self.df["scaled_NIR_2"][i]]))
        self.inputs = np.asarray(self.inputs)

Step 10: Training the Model (ANN) on the Ripening Stage Classes Based on Spectral Color

After preprocessing and scaling (normalizing) the ripening stages data set to create inputs and labels, I split them as training (95%) and test (5%) sets:

    def split_data(self):
        l = len(self.df)
        # (95%, 5%) - (training, test)
        self.train_inputs = self.inputs[0:int(l*0.95)]
        self.test_inputs = self.inputs[int(l*0.95):]
        self.train_labels = self.labels[0:int(l*0.95)]
        self.test_labels = self.labels[int(l*0.95):]

Then, I built my artificial neural network (ANN) model by using Keras and trained it with the training set for nineteen epochs.

You can inspect these tutorials to learn about activation functions, loss functions, epochs, etc.

    def build_and_train_model(self):
        # Build the neural network:
        self.model = keras.Sequential([
            keras.Input(shape=(10,)),
            keras.layers.Dense(16),
            keras.layers.Dense(32),
            keras.layers.Dense(64),
            keras.layers.Dense(128),
            keras.layers.Dense(256),
            keras.layers.Dense(4, activation='softmax')
        ])
        # Compile:
        self.model.compile(optimizer='adam', loss="sparse_categorical_crossentropy", metrics=['accuracy'])
        # Train:
        self.model.fit(self.train_inputs, self.train_labels, epochs=19)
		
		...

After training with the training set (inputs and labels), the accuracy of my neural network model is between 0.82 and 0.89.

Step 11: Evaluating the Model

After building and training my artificial neural network model, I tested its accuracy and validity by utilizing the testing set (inputs and labels).

The evaluated accuracy of the model is 0.9412.

        ...
		
		# Test the accuracy:
        print("\n\nModel Evaluation:")
        test_loss, test_acc = self.model.evaluate(self.test_inputs, self.test_labels) 
        print("Evaluated Accuracy: ", test_acc)

After evaluating my neural network model, I saved it as a TensorFlow Keras H5 model (ANN_Ripeness_Detection.h5).

    def save_model(self):  
       self.model.save("E:\PYTHON\Vegetables_and_Fruits_Ripeness_Detection\model\ANN_Ripeness_Detection.h5")

Step 12: Experimenting With the Model by Making Predictions on Ripening Stages

After completing and saving my neural network model, I acquired spectral color data of fruits and vegetables with the device and formatted them as input parameters, depending on how I scaled (normalized) them (Step 5.2) while training the model. Then, I created a NumPy array (prediction_array) consisting of the formatted input parameters to predict ripening stages by spectral color with the model.

The model predicts possibilities of labels (ripeness classes) for each input as an array of 4 numbers. They represent the model's "confidence" that the given input array corresponds to each of the four different ripeness classes based on spectral color [0 - 3], as shown in Step 5.

  • [0] Early Ripe
  • [1] Partially Ripe
  • [2] Ripe
  • [3] Decay

⭐ In the make_predictions function:

⭐ Load the saved model (ANN_Ripeness_Detection.h5) into memory.

⭐ Format the input data.

⭐ Then, set the input tensor values with the formatted input data in a NumPy array for each fruit and vegetable input - prediction_array.

⭐ Run the model to make predictions.

⭐ Finally, display the most accurate label (ripeness class) predicted by the model for each input using the argmax() function.

    def make_predictions(self):
        saved_model = keras.models.load_model("E:\PYTHON\Vegetables_and_Fruits_Ripeness_Detection\model\ANN_Ripeness_Detection.h5")
        prediction_array = np.array([
            [0.131,0.148,0.184,0.765,1,1,1,0.809,0.276,0.273],
            [0.143,0.122,0.157,0.228,1,1,1,1,0.335,0.334],
            [0.038,0.032,0.032,0.030,0.042,0.055,0.467,0.880,0.100,0.102],
            [0.002,0.001,0.001,0.002,0.003,0.003,0.002,0.007,0.010,0.012]
        ])
        predictions = saved_model.predict(prediction_array)
        print("\n\nModel Predictions:\n")
        for i in range(len(prediction_array)):
            print("INPUT[" + str(i+1) + "] => ", self.ripeness_class_names[np.argmax(predictions[i])])

I run the model to predict the ripening stages of lots of fruits and vegetables at my disposal. As far as my experiment goes, the model is working impeccably :)

Step 13: Connections and Adjustments

// Connections
// Arduino Nano 33 IoT :           
//                                AS7341 11-Channel Spectral Color Sensor
// 3.3V --------------------------- +
// GND  --------------------------- -
// A5   --------------------------- C
// A4   --------------------------- D
//                                10K Potentiometer
// A0   --------------------------- S
//                                Class Button_1 (6x6)
// D2   ---------------------------
//                                Class Button_2 (6x6)
// D3   ---------------------------
//                                Class Button_3 (6x6)
// D4   ---------------------------
//                                Class Button_4 (6x6)
// D5   ---------------------------
//                                5mm Green LED
// D6   ---------------------------
//                                5mm Red LED
// D7   ---------------------------

To collect and send spectral color data generated by the AS7341 visible light sensor, I attach the sensor to an Arduino Nano 33 IoT. Then, I connected four class buttons to assign a ripeness class (Early Ripe, Partially Ripe, Ripe, Decay) when sending data to the PHP web application. I used a potentiometer (long-shaft with knob) to adjust the brightness of the built-in LED on the AS7341 sensor. Finally, I added 5mm green and red LEDs indicating the response from the web application - successful or not.

After completing connections, I fastened breadboards and the AS7341 sensor to an old bookshelf with a hot glue gun to create a solid device.

Step 14: Videos and Conclusion

After completing all steps above and experimenting, the device became a skillful assistant to categorize fruits and vegetables depending on ripening stages by spectral color :)

Step 15: Further Discussions

By bringing this ripeness detection method by spectral color into common usage, it can help to[1]:

🍒🥕 avert farmers from consuming a lot of time for manual sorting and examining of fruits and vegetables from harvest till its growth period,

🍒🥕 improve yield estimation,

🍒🥕 accelerate disease detection.

References

[1]Mrs Rex Fiona et al 2019 J. Phys.: Conf. Ser. 1362 012033.

Step 16: Schematics

Step 17: Code and Downloads

Install and inspect code files.

Download files.