Background

In this article, we will use softmax classifier to build a simple image classification neural network with an accuracy of 32%. In a Softmax classifier, binary logic is generalized and regressed to multiple logic. Softmax classifier will output the probability of the corresponding category.

We will first define a softmax classifier, then use the training set of CIFAR10 to train the neural network, and finally use the test set to verify the accuracy of the neural network.

Let’s get started.

Import dependencies

Like the previous course GettingStarted, we need to introduce each class of DeepLearning.scala.

In [1]:
import $ivy.`org.nd4j::nd4s:0.8.0`
import $ivy.`org.nd4j:nd4j-native-platform:0.8.0`
import $ivy.`com.chuusai::shapeless:2.3.2`
import $ivy.`org.rauschig:jarchivelib:0.5.0`
import $ivy.`com.thoughtworks.deeplearning::plugins-builtins:2.0.0`
import $ivy.`org.plotly-scala::plotly-jupyter-scala:0.3.2`
import $ivy.`com.thoughtworks.each::each:3.3.1`
import $plugin.$ivy.`org.scalamacros:paradise_2.11.11:2.1.0`

import scala.concurrent.ExecutionContext.Implicits.global
import org.nd4j.linalg.api.ndarray.INDArray
import org.nd4j.linalg.factory.Nd4j
import com.thoughtworks.deeplearning.plugins.Builtins
import com.thoughtworks.feature.Factory
import plotly._
import plotly.element._
import plotly.layout._
import plotly.JupyterScala._
import com.thoughtworks.future._
import scala.concurrent.Await
import scala.concurrent.duration.Duration
import com.thoughtworks.each.Monadic._
import scalaz.std.stream._
Out[1]:
import $ivy.$                     

import $ivy.$                                    

import $ivy.$                             

import $ivy.$                               

import $ivy.$                                                      

import $ivy.$                                             

import $ivy.$                                  

import $plugin.$                                            


import scala.concurrent.ExecutionContext.Implicits.global

import org.nd4j.linalg.api.ndarray.INDArray

import org.nd4j.linalg.factory.Nd4j

import com.thoughtworks.deeplearning.plugins.Builtins

import com.thoughtworks.feature.Factory

import plotly._

import plotly.element._

import plotly.layout._

import plotly.JupyterScala._

import com.thoughtworks.future._

import scala.concurrent.Await

import scala.concurrent.duration.Duration

import com.thoughtworks.each.Monadic._

import scalaz.std.stream._

To reduce the line numbers outputted by jupyter-scala and to make sure that the page output will not be too long, we need to set pprintConfig.

In [2]:
pprintConfig() = pprintConfig().copy(height = 2)

Build your own neural network.

Set learning rate

Learning rate need to be set for the full connection layer. Learning rate visually describes the change rate of weight. A too-low learning rate will result in slow decrease of loss, which will require longer time for training; A too-high learning rate will result in rapid decrease of loss at first while fluctuation around the lowest point afterward.

In [4]:
val INDArrayLearningRatePluginUrl = "https://gist.githubusercontent.com/TerrorJack/118487016d7973d67feb489449dee156/raw/778bb1b68a664c752b0945111220326731310214/INDArrayLearningRate.sc"
interp.load(scala.io.Source.fromURL(new java.net.URL(INDArrayLearningRatePluginUrl)).mkString)
Out[4]:
INDArrayLearningRatePluginUrl: String = "https://gist.githubusercontent.com/TerrorJack/118487016d7973d67feb489449dee156/raw/778bb1b68a664c752b0945111220326731310214/INDArrayLearningRate.sc"
In [6]:
// `interp.load` is a workaround for https://github.com/lihaoyi/Ammonite/issues/649 and https://github.com/scala/bug/issues/10390
interp.load("""
  val hyperparameters = Factory[Builtins with INDArrayLearningRate].newInstance(learningRate = 0.1)
""")

Write softmax

To use softmax classifier (softmax classifier is a neural network combined by softmax and a full connection), we first need to write softmax function, formula:

In [7]:
import hyperparameters.implicits._
Out[7]:
import hyperparameters.implicits._
In [8]:
import hyperparameters.INDArrayLayer

def softmax(scores: INDArrayLayer): INDArrayLayer = {
  val expScores = hyperparameters.exp(scores)
  expScores / expScores.sum(1)
}
Out[8]:
import hyperparameters.INDArrayLayer


defined function softmax

Compose your neural network

Define a full connection layer and initialize Weight, Weight shall be a two-dimension INDArray of NumberOfPixels × NumberOfClasses. scores is the score of each image corresponding to each category, representing the feasible probability of each category corresponding to each image.

In [9]:
//10 label of CIFAR10 images(airplane,automobile,bird,cat,deer,dog,frog,horse,ship,truck)
val NumberOfClasses: Int = 10
val NumberOfPixels: Int = 3072
Out[9]:
NumberOfClasses: Int = 10
NumberOfPixels: Int = 3072
In [10]:
import hyperparameters.INDArrayWeight

val weight = {
    import org.nd4s.Implicits._
    INDArrayWeight(Nd4j.randn(NumberOfPixels, NumberOfClasses) * 0.001)
}

def myNeuralNetwork(input: INDArray): INDArrayLayer = {
    softmax(input.dot(weight))
}
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Out[10]:
import hyperparameters.INDArrayWeight


weight: Object with hyperparameters.INDArrayWeightApi with hyperparameters.WeightApi with hyperparameters.WeightApi = Weight[fullName=$sess.cmd9Wrapper.Helper.weight]
defined function myNeuralNetwork

Create LossFunction

To learn about the prediction result of the neural network, we need to write the loss function lossFunction. We use cross-entropy loss to make comparison between this result and the actual result before return the score. Formula:

In [11]:
import hyperparameters.DoubleLayer

def lossFunction(input: INDArray, expectOutput: INDArray): DoubleLayer = {
    val probabilities = myNeuralNetwork(input)
    -(hyperparameters.log(probabilities) * expectOutput).mean
}
Out[11]:
import hyperparameters.DoubleLayer


defined function lossFunction

Prepare data

Read data

To read the images and corresponding label information for test data from CIFAR10 database and process them, we need import $file.ReadCIFAR10ToNDArray. This is a script file containing the read and processed CIFAR10 data, provided in this course.

In [12]:
import $url.{` https://raw.githubusercontent.com/ThoughtWorksInc/DeepLearning.scala-website/v1.0.0-doc/ipynbs/ReadCIFAR10ToNDArray.sc` => ReadCIFAR10ToNDArray}

val trainNDArray = ReadCIFAR10ToNDArray.readFromResource("/cifar-10-batches-bin/data_batch_1.bin", 1000)

val testNDArray = ReadCIFAR10ToNDArray.readFromResource("/cifar-10-batches-bin/test_batch.bin", 100)
Compiling https://raw.githubusercontent.com/ThoughtWorksInc/DeepLearning.scala-website/v1.0.0-doc/ipynbs/ReadCIFAR10ToNDArray.sc
downloading data...
unzip file...
download and unzip done.
Out[12]:
import $url.$                                                                                                                                           


trainNDArray: shapeless.::[INDArray, shapeless.::[INDArray, shapeless.HNil]] = [[0.23, 0.17, 0.20, 0.27, 0.38, 0.46, 0.54, 0.57, 0.58, 0.58, 0.51, 0.49, 0.55, 0.56, 0.54, 0.50, 0.54, 0.52, 0.48, 0.54, 0.54, 0.52, 0.53, 0.54, 0.59, 0.64, 0....
testNDArray: shapeless.::[INDArray, shapeless.::[INDArray, shapeless.HNil]] = [[0.62, 0.62, 0.64, 0.65, 0.62, 0.61, 0.63, 0.62, 0.62, 0.62, 0.63, 0.62, 0.63, 0.65, 0.66, 0.66, 0.65, 0.63, 0.62, 0.62, 0.61, 0.58, 0.59, 0.58, 0.58, 0.56, 0....

Process data

Before passing data to the softmax classifier, we first process label data with (one hot encoding): transform INDArray of NumberOfPixels × 1 into INDArray of NumberOfPixels × NumberOfClasses. The value of correct classification corresponding to each line is 1, and the values of other columns are 0. The reason for differentiating the training set and test set is to make it clear that whether the network is over trained which leads to overfitting. While processing label data, we used Utils, which is also provided in this course.

In [13]:
val trainData = trainNDArray.head
val testData = testNDArray.head

val trainExpectResult = trainNDArray.tail.head
val testExpectResult = testNDArray.tail.head

import $url.{`https://raw.githubusercontent.com/ThoughtWorksInc/DeepLearning.scala-website/v1.0.0-doc/ipynbs/Utils.sc` => Utils}

val vectorizedTrainExpectResult = Utils.makeVectorized(trainExpectResult, NumberOfClasses)
val vectorizedTestExpectResult = Utils.makeVectorized(testExpectResult, NumberOfClasses)
Compiling https://raw.githubusercontent.com/ThoughtWorksInc/DeepLearning.scala-website/v1.0.0-doc/ipynbs/Utils.sc
Out[13]:
trainData: INDArray = [[0.23, 0.17, 0.20, 0.27, 0.38, 0.46, 0.54, 0.57, 0.58, 0.58, 0.51, 0.49, 0.55, 0.56, 0.54, 0.50, 0.54, 0.52, 0.48, 0.54, 0.54, 0.52, 0.53, 0.54, 0.59, 0.64, 0....
testData: INDArray = [[0.62, 0.62, 0.64, 0.65, 0.62, 0.61, 0.63, 0.62, 0.62, 0.62, 0.63, 0.62, 0.63, 0.65, 0.66, 0.66, 0.65, 0.63, 0.62, 0.62, 0.61, 0.58, 0.59, 0.58, 0.58, 0.56, 0....
trainExpectResult: INDArray = [6.00, 9.00, 9.00, 4.00, 1.00, 1.00, 2.00, 7.00, 8.00, 3.00, 4.00, 7.00, 7.00, 2.00, 9.00, 9.00, 9.00, 3.00, 2.00, 6.00, 4.00, 3.00, 6.00, 6.00, 2.00, 6.00, 3.0...
testExpectResult: INDArray = [3.00, 8.00, 8.00, 0.00, 6.00, 6.00, 1.00, 6.00, 3.00, 1.00, 0.00, 9.00, 5.00, 7.00, 9.00, 8.00, 5.00, 7.00, 8.00, 6.00, 7.00, 0.00, 4.00, 9.00, 5.00, 2.00, 4.0...
import $url.$                                                                                                             


vectorizedTrainExpectResult: INDArray = [[0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 1.00, 0.00, 0.00, 0.00],
 [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 1.00],
...
vectorizedTestExpectResult: INDArray = [[0.00, 0.00, 0.00, 1.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00],
 [0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 1.00, 0.00],
...

Train your neural network

To observe the training process of the neural network, we need to output loss; while training the neural network, the loss shall be decreasing.

In [14]:
var lossSeq: IndexedSeq[Double] = IndexedSeq.empty

@monadic[Future]
val trainTask: Future[Unit] = {
  val lossStream = for (_ <- (1 to 2000).toStream) yield {
    lossFunction(trainData, vectorizedTrainExpectResult).train.each
  }
  lossSeq = IndexedSeq.concat(lossStream)
}
Out[14]:
lossSeq: IndexedSeq[Double] = Vector()
trainTask: Future[Unit] = scalaz.IndexedContsT@25ef3ca5

Predict your Neural Network

We use the processed test data to verify the prediction result of the neural network and compute the accuracy. The accuracy shall be about 32%.

In [15]:
Await.result(trainTask.toScalaFuture, Duration.Inf)
In [16]:
val predictResult = Await.result(myNeuralNetwork(testData).predict.toScalaFuture, Duration.Inf)
println("The accuracy is " + Utils.getAccuracy(predictResult,testExpectResult) + "%")
The accuracy is 32.0%
Out[16]:
predictResult: INDArray = [[0.03, 0.05, 0.17, 0.13, 0.01, 0.13, 0.42, 0.00, 0.04, 0.00],
 [0.03, 0.17, 0.00, 0.05, 0.00, 0.01, 0.00, 0.00, 0.18, 0.55],
...
In [17]:
plotly.JupyterScala.init()
Seq(Scatter(lossSeq.indices, lossSeq)).plot(title = "loss by time")
Out[17]:
res16_1: String = "plot-1479964318"

Summary

We have learned the follows in this article:

  • Prepare and process CIFAR10 data
  • Write softmax classifier
  • Use the prediction image of the neural network written by softmax classifier to match with the probability of each category.

Download this tutorial

DeepLearning.scala is an open source deep-learning toolkit in Scala created by our colleagues at ThoughtWorks. We're excited about this project because it uses differentiable functional programming to create and compose neural networks; a developer simply writes code in Scala with static typing.