DL4J: How to create a neural network that draws images – Step by step guide

DL4J: How to create a neural network that draws images – Step by step guide

Neural networks, machine learning, artificial intelligence – I get the impression that these slogans attack us from everywhere. They are mainly associated with the giants of the IT industry, who from time to time report spectacular progress in this field. I decided to dispel myths about machine learning using a series of articles explaining this problem by interesting examples. In this article, I will show you how to build and learn a neural network to generate one image based on another step by step.

Agenda
  1. What will you learn from this guide?
  2. Implementation of the application and the GUI
  3. Modeling of the neural network
  4. Preparation of input data
  5. Learning the neural network
  6. Application test

 

Application sources

The source codes for the application described in this guide can be found in the 01-neural-network-painter directory located in the repository:

 

Additional resources:

 

1. What will you learn from this guide?

In this guide we will create a neural network, which we will put the task of transforming one image into another. The application will be built on the basis of the DL4J application template described in the previous article based on JavaFx and Spring Boot. Thanks to the fact that we will use it, the amount of code necessary to implement has been reduced to only one controller and one form.

Despite the small amount of code, we will be able to get through the most important issues of machine learning. We will start the journey by preparing training and test data. In the following, we will create a neural network model fitted to our needs, which we will then learn to solve the task set for it. Finally, we ask the network for the result that we will visualize as a picture.

The applied template of the application allows us to track the progress of learning, and to visualize the current result of the network during the learning process. Below is a screenshot of the application after doing nearly 100,000 iterations of the learning process.

 

If you have not read about the application template yet, now is the right time to do it:

 

2. Implementation of the application and the GUI

We will build our application on the previously mentioned template, therefore download it from the repository, unpack it and then import the project by opening the build.gradle file in the preferred IDE(I recommend IntelliJ IDEA). You can change the name of the project by editing settings.gradle file.

 

 

The next step is to implement the missing part of the GUI. In the case of this article, it will be a form consisting of three pictures. The first is the picture that the neural network receives at the input. The second is the network response (result) transformed into an image. The third, however, is the image we expected.

However, before we implement the form, prepare or download three 100x100px images and put them in the src/main/resources/data directory. Name them input.png, expected.png and test.png. We use the test.png image to check the response of a neural network to a different image on the input than the learned one. You can download images from the repository.

 

We implement the GUI by editing the fxml/custom-learning-gui.fxml file. My implementation looks like the listing below. Because the 100x100px graphics is quite small on the Full HD screen of the computer, I decided to enlarge it three times. It is worth paying attention to a few elements here:

  • The form has a defined controller that comes from the template: fx:controller="com.bettercoding.dl4j.controller.CustomLearningGuiControllerImpl"
  • The controls you want to use in the controller are marked with the attribute fx:id. For example fx:id="inputImageView".

 

It is time to initialize the defined images in the controller by editing the CustomLearningGuiControllerImpl class:

 

Running the com.bettercoding.dl4j.Dl4JApp application should give the result like below:

 

3. Modeling of the neural network

The moment has come when we are beginning to design and implement the first neural network. However, before we do it, it will be necessary to get the basics about its construction.


Neural networks were initially modeled on the action of nerve cells. In my opinion and it is still a fairly good analogy to imagining the principles of operation and the processes that take place in them. The basic element of the neural network is the neuron. In the simplest form it consists of inputs, weights, adder, activation function and output. Inputs and their weights are the equivalent of dendrites in the nerve cell. The adder and activation function play a role analogous to the cell nucleus, while the neuron output is the equivalent of axons.

 

Signals flowing from inputs are multiplied by the weight corresponding to them and summed up. Then the result of the activation function for the sum is calculatead. Such a function may be, for example, a linear or sigmoidal function.

 

Teaching a neuron involves sensitizing it to a signal coming from one of the inputs by modifying the weight of this input, which affects the participation of this input in total sum. For example, if we increase the weight value of the first input by ten times, the share of this signal will also increase, which in turn will weaken the reaction to the remaining inputs.

If we put the neurons in layers so that we connect the outputs of the neurons with the neuron inputs of the next layer, then we get a neural network. Below is presented a three-layer neural network with three input neurons, two in the hidden layer, and the only one in the output layer.

 


For the purposes of this article, we will implement a neural network consisting of four hidden layers and one output layer. We will use two different activation functions: Activation.IDENTITY(liniowej) and Activation.LEAKYRELU. The network has 5 input neurons meaning the x and y pixel coordinates of the input image, and its color splitted into red, green and blue values. The output layer has 3 neurons which are the color of the output pixel, which corresponds to the position of the input pixel.

 

We implement the described neural network by adding the following code into the CustomLearningGuiControllerImpl class:

 

4. Preparation of input data

It’s time to prepare examples on the basis of which we will learn our network of tasks set before it. In the case of this article, it will be a collection of vectors (one for each pixel, i.e. a total of 10,000) consisting of 5 features (x,y,r,g,b) and the expected result – 3 labels (r,g,b). DL4j provides the data structure that is used for this purpose, namely DataSet.

 

We convert the image into DataSet by the ImageUtils.convertToDataSet function as it is shown below. I will not discuss how we make this conversion, because the code contained in the repository is clear enough.

 

At this moment, we will also prepare data for the purposes of the test, which we will perform every five seconds. It will consist in converting the network result for all points of the input image, and then composing the resulting image from the series of neural network responses. Therefore, we will need a variable:

 

5. Learning the neural network


By the process of learning, I will pass over briefly, because its detailed discussion requires deepening knowledge of topics that we have not discussed yet. At the moment, you need to know that we use the backpropagation algorithm. The implementation of the network learning process consists in selecting a sample of 1000 points, and then starting a single loop of learning process by calling neuralNetwork.fit method, as it is shown below:

 

In addition, the framework prepared by me refreshes the GUI every 5s calling the onRefreshGUI method. We can use it to convert all points of the output image by the network and display them on the GUI. Thanks to this, we can observe the progress of teaching the neural network.

 

The last method to implement is to handle the Test button. The following code is an implementation that loads the test.png file, then it is processed by the neural network and displayed in the outputImage control.

 

6. Application test

You can download the finished application from the repository. Below are a few screenshots showing how the process of learning the network looked like depending on the number of train loops.

 

If you think this post is valuable, please leave me +1 or just share it. This action will allow me to reach a wider audience.
Thank You in advance.

 

2
Leave a Reply

avatar
1 Comment threads
1 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
2 Comment authors
lukasz.cieslabanq Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
banq
Guest
banq

Could not resolve all files for configuration ‘:compileClasspath’.
> Could not find nd4j-native.jar (org.nd4j:nd4j-native:1.0.0-beta2).

Close Menu