YOLO v3 — From Python To Java ?

Engineering
Sabin
by Sabin 

“YOLO” this word is something special. If you ask a general person the meaning of this word, you would probably get something like “YOU ONLY LIVE ONCE” but if you ask a nerd like me you might get “YOU ONLY LOOK ONCE”, contradiction of meaning?

Its a matter of domains that determines the meaning of this word. Never the less in this article I will be referencing this as “YOU ONLY LOOK ONCE”. You might be pondering what does this mean. In the field of AI, YOLO is a state of the art object detection algorithm.

Output Of Yolov3

Output Of YOLO v3

There have been various iteration of this algorithm but in this article I will be focusing on the v3 implementation from Darknet. To learn more about it’s working principles and how the algorithm localizes object present in the image visit darknet’s homepage.

But in brief we provide an image to the neural network , it divides image into various regions where object were detected with boxes around them, the boxes having higher score are show and boxes having low score are suppressed aka non max suppression.

Motivation

Everyone knows python for its popularity in AI community. But there are various other languages that we can implement this algorithm. There may not be a lot of supporting community for other languages but there are implementation of Open-CV , TensorFlow etc. for other languages too. So, I wanted to try how difficult it would be to implement simple YOLO v3 algorithm in other language.

Same as before this article focuses more on the understanding of how the implementation flow rather than explaining the nitty-gritty part of YOLO.

I will be showing and explaining both implementation i.e in Java and python.

Resources

As always, You can find this implementation on my personal GitHub repository GitHub : Download Code

Darknet : Download Weights ,Config Files and Test Images

I have included few of the test images in my repository but feel free to do your thing. Since, weight files are big I have not included them on the GitHub repository. You can download the weights form the above mentioned darknets yolo’s home page.

Python Implementation

So lets get started with the implementation. If you want to see the Java’s Implementation scroll down to Java Implementation.

Requirement: check requirements.txt

Importing Libraries

import cv2
import numpy as np
import os
from datetime import datetime
from pathlib import Path

We will be using open cv’s dnn to load the neural network which will be the common ground for both implementation.

Defining Some Global

current = os.path.dirname(os.path.abspath(__file__))
parent = Path(current).parent
yolo_weights = f'{parent}/Assets/models/yolov3.weights'
yolo_cfg = f'{parent}/Assets/models/yolov3.cfg'
coco_names = f'{parent}/Assets/models/coco.names'

Here we define where the weights , config , classes names are located. Since, there are 80 types of classes it’s better to load this from a file rather than creating an array of gazillion length.

Loading YOLO

def load_yolo():
    net = cv2.dnn.readNet(yolo_weights, yolo_cfg)
    classes = []

    with open(coco_names, "r") as f:
        classes = [line.strip() for line in f.readlines()]

    layers_names = net.getLayerNames()
    unconnected_layer = net.getUnconnectedOutLayers()
    output_layers = [layers_names[i - 1] for i in unconnected_layer]
    colors = np.random.uniform(0, 255, size=(len(classes), 3))
    return net, classes, colors, output_layers

In this function, We take help from OpenCv’s DNN library which facilitates us for loading a neural network from files. The neural network structure is defined on yolov3.cfg and all the weights required to initialize every layer is defined on yolv3.weights.

Along with that we load the number of classes for classification which is 80 classes. We read that from a file for obvious reasons. Lastly, we need reference from where the neural network takes its input and spit out the outputs. This work could be very tedious but thanks to opencv we could do this with few lines of code. We can get the names of all the layers in NN from getLayerNames() and get all the unconnected layers from function namely getUnconnectedOutLayers().

Loading Images

def load_image(path):
    image = cv2.imread(path)
    image = cv2.resize(image, None, fx=0.9, fy=0.9)
    height, width, channels = image.shape
    return image, height, width, channels

So, for a neural network to get some input we need to load the image in a format that a neural network can understand which is of course numbers.

Again , We load the image with the help of OpenCv’s imread function and do some resizing.

Detecting Objects

def detect_object(img, net, outputLayers):
    blob = cv2.dnn.blobFromImage(img, scalefactor=0.00392, size=(608, 608), mean=(0, 0, 0), swapRB=True, crop=False)
    net.setInput(blob)
    outputs = net.forward(outputLayers)
    return blob, outputs

As mentioned previously we provide an image to the neural network and the network divides the images into different small regions for where the object are located. So we take the image that we have loaded previously, then do some pre-processing since our NN architecture only takes image of 608x608 size.

After, We resize the image into the size that our NN can take , we feed it through the NN and the NN spits out the result in a form of confidence, bounding boxes and score of each boxes. More on that later

Getting Box Dimensions

def get_box_dimensions(outputs, height, width):
    boxes = []
    confs = []
    class_ids = []

    for output in outputs:
        for detect in output:
            scores = detect[5:]
            class_id = np.argmax(scores)
            conf = scores[class_id]
            if conf > 0.5:
                center_x = int(detect[0] * width)
                center_y = int(detect[1] * height)
                w = int(detect[2] * width)
                h = int(detect[3] * height)
                x = int(center_x - w / 2)
                y = int(center_y - h / 2)
                boxes.append([x, y, w, h])
                confs.append(float(conf))
                class_ids.append(class_id)

    return boxes, confs, class_ids

After our NN have given an output we need to manually define what kind of boxes could be shown or more over set the threshold of the detected object. Generally YOLO detects everything its our job to filter the object detected. So we iterate over the outputs and here we only take the object that have confidence level of more than 50% i.e 0.5. So all the objects with confidence level greater than 0.5 are added and array of boxes are returned with its confidence level and the category of the object namely class id.

Drawing Labels

def draw_labels(boxes, confs, colors, class_ids, classes, img):
    indexes = cv2.dnn.NMSBoxes(boxes, confs, 0.5, 0.4)
    font = cv2.FONT_HERSHEY_PLAIN
    for i in range(len(boxes)):
        if i in indexes:
            x, y, w, h = boxes[i]
            label = str(classes[class_ids[i]])
            color = colors[i]
            cv2.rectangle(img, (x, y), (x + w, y + h), color, 2)
            cv2.putText(img, label, (x, y - 5), font, 1, color, 1)
    cv2.imwrite(f'{parent}/outputimages/python/{datetime.now()}.jpg.webp', img)

Difficult part of implementation is kinda over now we just iterate over the filtered objects and set a threshold for the non max suppression. Since same object can be detected by different bounding boxes, we set a threshold to say that okay if the boxes have score of less than something(some value) then treat it as already detected by another box and ignore it.

Lastly, we put a rectangle over the box after suppression , label it with the name of the class nothing fancy there.

Creating A Pipeline

def image_detect(img_path):
    model, classes, colors, output_layers = load_yolo()
    image, height, width, channels = load_image(img_path)
    blob, outputs = detect_object(image, model, output_layers)
    boxes, confs, class_ids = get_box_dimensions(outputs, height, width)
    draw_labels(boxes, confs, colors, class_ids, classes, image)

Nothing fancy here just a function to wrap all the work in a sequential order.

Java Implementation

Since Java is a typed languages there are grater level of complexities. Firstly, there are no tools to download OpenCV’s dependencies. You have to download the .jar file and set it to the path variable manually . In Java referencing multi dimensional arrays are demanding. Never the less lets, get started.

Importing Libraries

In comparison to python there are more lines of code to be written in java(*Why Java Why).

import org.opencv.core.*;
import org.opencv.dnn.Dnn;
import org.opencv.dnn.Net;
import org.opencv.imgcodecs.Imgcodecs;
import org.opencv.imgproc.Imgproc;

import java.io.File;
import java.io.FileNotFoundException;
import java.util.ArrayList;
import java.util.List;
import java.util.Scanner;

Here we import the OpenCv’s DNN library after configuring our project to detect opencv.jar file as a dependencies.

Since Java follows Class and Object relation , we will be working with a class namely Vision for the sake of simplicity.

Defining Some Globals

private final String model_weights;
private final String model_config;
private final String current_dir;
private final String class_file_name_dir;
private final String output_path;
private final List<String> classes;
private final List<String> output_layers;
private String input_path;
private List<String> layer_names;
private Net network;
private Size size;
private Integer height;
private Integer width;
private Integer channels;
private Scalar mean;
private Mat image;
private Mat blob;
private List<Mat> outputs;
private List<Rect2d> boxes;
private List<Float> confidences;
private List<Integer> class_ids;
private String outputFileName;
private boolean save;
private boolean errors;

That’s a lot of variables with global scope. Yes, but never-mind.

Defining Constructor

public Vision(String inputPath, String outputPath, Integer image_size, String outputFileName) {
    System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
    this.input_path = inputPath;
    this.output_path = outputPath;
    this.outputFileName = outputFileName;
    boxes = new ArrayList<>();
    classes = new ArrayList<>();
    class_ids = new ArrayList<>();
    layer_names = new ArrayList<>();
    confidences = new ArrayList<>();
    double[] means = {0.0, 0.0, 0.0};
    mean = new Scalar(means);
    output_layers = new ArrayList<>();
    size = new Size(image_size, image_size);
    current_dir = System.getProperty("user.dir");
    model_weights = current_dir + "/Assets/models/yolov3-608.weights";
    model_config = current_dir + "/Assets/models/yolov3-608.cfg";
    class_file_name_dir = current_dir + "/Assets/models/coco.names";
    save = true;

}

Since, Java fancies class and object, we initialize some variables while creating an instance of the class so that it can be accessed in the future.

Here, as usual we define the location of the config, weights and class names.

Implementing Some Non Python Function

private static int argmax(List<Float> array) {
    float max = array.get(0);
    int re = 0;
    for (int i = 1; i < array.size(); i++) {
        if (array.get(i) > max) {
            max = array.get(i);
            re = i;
        }
    }
    return re;
}

In python, numpy provided this function as an inbuilt , not same applies for Java(*ugh).

Loading Classes

private void setClasses() {
    try {
        File f = new File(class_file_name_dir);
        Scanner reader = new Scanner(f);
        while (reader.hasNextLine()) {
            String class_name = reader.nextLine();
            classes.add(class_name);
        }
    } catch (FileNotFoundException e) {
        errors = true;
    }
}

Nothing fancy just huge lines of code to read a file.

Loading Network

private void setNetwork() {
    network = Dnn.readNet(model_weights, model_config);
}

Same as before , Loading the neural network form config and weight file. Thanks to OpenCv’s DNN library.

Setting Unconnected Layers

private void setUnconnectedLayers() {

    for (Integer i : network.getUnconnectedOutLayers().toList()) {
        output_layers.add(layer_names.get(i - 1));
    }
}

Setting Layer Names

private void setLayerNames() {
    layer_names = network.getLayerNames();
}

Again nothing extraordinary , all we did is split the process of setting unconnected layer and setting layer names.

Loading Image

private void loadImage() {
    Mat img = Imgcodecs.imread(input_path);
    Mat resizedImage = new Mat();
    Imgproc.resize(img, resizedImage, size, 0.9, 0.9);
    height = resizedImage.height();
    width = resizedImage.width();
    channels = resizedImage.channels();
    image = resizedImage;
}

Here comes the interesting part. In python, we had array manipulation library like numpy , but in Java similar concept is reflected with Mat(). In brief it is numpy for Java(*lol).

Mat() are considered to be faster than referencing a normal multi dimensional array.

Detecting Objects

private void detectObject() {
    Mat blob_from_image = Dnn.blobFromImage(image, 0.00392, size, mean, true, false);
    network.setInput(blob_from_image);
    outputs = new ArrayList<Mat>();
    network.forward(outputs, output_layers);
    blob = blob_from_image;
}

This might look familiar to python’s implementation. All we did was read an image (in Mat() or Matrix format),feed it through the neural network and Saved the output of the NN to another matrix variable.

Getting Box Dimensions

private void getBoxDimensions() {
    for (Mat output : outputs) {

        for (int i = 0; i < output.height(); i++) {
            Mat row = output.row(i);
            MatOfFloat temp = new MatOfFloat(row);
            List<Float> detect = temp.toList();
            List<Float> score = detect.subList(5, 85);
            int class_id = argmax(score);
            float conf = score.get(class_id);
            if (conf >= 0.4) {
                int center_x = (int) (detect.get(0) * width);
                int center_y = (int) (detect.get(1) * height);
                int w = (int) (detect.get(2) * width);
                int h = (int) (detect.get(3) * height);
                int x = (center_x - w / 2);
                int y = (center_y - h / 2);
                Rect2d box = new Rect2d(x, y, w, h);
                boxes.add(box);
                confidences.add(conf);
                class_ids.add(class_id);

            }

        }

    }
}

The intuition of getting the boxes dimensions are the same, only the length of the code and data structures are different.

Drawing Labels

private void drawLabels() {
    double[] rgb = new double[]{255, 255, 0};
    Scalar color = new Scalar(rgb);
    MatOfRect2d mat = new MatOfRect2d();
    mat.fromList(boxes);
    MatOfFloat confidence = new MatOfFloat();
    confidence.fromList(confidences);
    MatOfInt indices = new MatOfInt();
    int font = Imgproc.FONT_HERSHEY_PLAIN;
    Dnn.NMSBoxes(mat, confidence, (float) (0.4), (float) (0.4), indices);
    List indices_list = indices.toList();
    for (int i = 0; i < boxes.size(); i++) {
        if (indices_list.contains(i)) {
            if (save) {
                Rect2d box = boxes.get(i);
                Point x_y = new Point(box.x, box.y);
                Point w_h = new Point(box.x + box.width, box.y + box.height);
                Point text_point = new Point(box.x, box.y - 5);
                Imgproc.rectangle(image, w_h, x_y, color);
                String label = classes.get(class_ids.get(i));
                Imgproc.putText(image, label, text_point, font, 1, color);
            }

        }
    }
    if (save) {
        Imgcodecs.imwrite(output_path + "\\" + outputFileName + ".png.webp", image);
    }

}

This might look complex but we are doing same amount of work that we did in python implementation. You can clearly see why python has easy learning curve. We can do far more work in less number of lines. No wonder why it is so popular , because let me tell you a secret “Programmers are lazy” (*lol).

Creating A Pipeline

public void loadPipeline() {
    try {
        setNetwork();
        setClasses();
        setLayerNames();
        setUnconnectedLayers();
        loadImage();
        detectObject();
        getBoxDimensions();
        drawLabels();
    } catch (Exception e) {
        errors = true;
    }

}

Finally, a public method that wraps all the work. It really amazed me how python can do much more with far less code. Contrary it might be slow but as we all know “In Java Program Runs Faster, But in Python Programmers Work Faster”.

OUTPUTS

Share Article

Few more blogs...