Background

In this article, we will build a three-layers model for visual recognition. The three-layers model contains a convolutional layer and two full connection layers.

We will use the training set of CIFAR10 to train the model, finally use the test set to verify the accuracy of the model.

The model will achieve 40% accuracy.

Let’s get started.

Import dependencies

Like the previous course GettingStarted, we need to introduce each class of DeepLearning.scala.

In [1]:
import $ivy.`org.nd4j::nd4s:0.8.0`
import $ivy.`org.nd4j:nd4j-native-platform:0.8.0`
import $ivy.`com.chuusai::shapeless:2.3.2`
import $ivy.`org.rauschig:jarchivelib:0.5.0`
import $ivy.`com.thoughtworks.deeplearning::plugins-builtins:2.0.1`
import $ivy.`org.plotly-scala::plotly-jupyter-scala:0.3.2`
import $ivy.`com.thoughtworks.each::each:3.3.1`
import $ivy.`com.thoughtworks.each:each_2.11:3.3.1`
import $plugin.$ivy.`org.scalamacros:paradise_2.11.11:2.1.0`

import plotly._
import plotly.element._
import plotly.layout._
import plotly.JupyterScala._
plotly.JupyterScala.init()

import org.nd4j.linalg.api.ndarray.INDArray
import org.nd4j.linalg.factory.Nd4j
import com.thoughtworks.deeplearning.DeepLearning
import com.thoughtworks.deeplearning.plugins._
import com.thoughtworks.feature.Factory
import com.thoughtworks.future._
import scala.concurrent.Await
import scala.concurrent.duration.Duration
import com.thoughtworks.each.Monadic._
import scalaz.std.stream._

import java.util.concurrent.Executors
import scala.concurrent.ExecutionContext
val singleThreadExecutor = Executors.newSingleThreadExecutor()
implicit val singleThreadExecutionContext = ExecutionContext.fromExecutor(singleThreadExecutor)
Out[1]:
import $ivy.$                     

import $ivy.$                                    

import $ivy.$                             

import $ivy.$                               

import $ivy.$                                                      

import $ivy.$                                             

import $ivy.$                                  

import $ivy.$                                      

import $plugin.$                                            


import plotly._

import plotly.element._

import plotly.layout._

import plotly.JupyterScala._

import org.nd4j.linalg.api.ndarray.INDArray

import org.nd4j.linalg.factory.Nd4j

import com.thoughtworks.deeplearning.DeepLearning

import com.thoughtworks.deeplearning.plugins._

import com.thoughtworks.feature.Factory

import com.thoughtworks.future._

import scala.concurrent.Await

import scala.concurrent.duration.Duration

import com.thoughtworks.each.Monadic._

import scalaz.std.stream._


import java.util.concurrent.Executors

import scala.concurrent.ExecutionContext

singleThreadExecutor: java.util.concurrent.ExecutorService = java.util.concurrent.Executors$FinalizableDelegatedExecutorService@3a733677
singleThreadExecutionContext: concurrent.ExecutionContextExecutor = scala.concurrent.impl.ExecutionContextImpl@6cc74c0f

To reduce the line numbers outputted by jupyter-scala and to make sure that the page output will not be too long, we need to set pprintConfig.

In [2]:
pprintConfig() = pprintConfig().copy(height = 2)

Build your model.

Set Plugins

Learning rate need to be set for the full connection layer. Learning rate visually describes the change rate of weight. A too-low learning rate will result in slow decrease of loss, which will require longer time for training; A too-high learning rate will result in rapid decrease of loss at first while fluctuation around the lowest point afterward.

In [4]:
val INDArrayLearningRatePluginUrl = "https://gist.githubusercontent.com/Rabenda/f06279e648e45bd574dc382abb4c44ac/raw/7bd7a871030988c58524108c5985f71002f82012/INDArrayLearningRate.sc"
interp.load(scala.io.Source.fromURL(new java.net.URL(INDArrayLearningRatePluginUrl)).mkString)
Out[4]:
INDArrayLearningRatePluginUrl: String = "https://gist.githubusercontent.com/Rabenda/f06279e648e45bd574dc382abb4c44ac/raw/7bd7a871030988c58524108c5985f71002f82012/INDArrayLearningRate.sc"
In [6]:
val CNNsPluginUrl = "https://gist.github.com/Atry/15b7d9a4c63d95ad3d67e94bf20b4f69/raw/59f7ee4dff0dde3753f560633574265e950edc93/CNN.sc"
interp.load(scala.io.Source.fromURL(new java.net.URL(CNNsPluginUrl)).mkString)
Out[6]:
CNNsPluginUrl: String = "https://gist.github.com/Atry/15b7d9a4c63d95ad3d67e94bf20b4f69/raw/59f7ee4dff0dde3753f560633574265e950edc93/CNN.sc"
In [8]:
val L2RegularizationPluginUrl = "https://gist.githubusercontent.com/TerrorJack/a60ff752270c40a6485ee787837390aa/raw/119cbacb29dc12d74ae676b4b02687a8f38b02e4/L2Regularization.sc"
interp.load(scala.io.Source.fromURL(new java.net.URL(L2RegularizationPluginUrl)).mkString)
Out[8]:
L2RegularizationPluginUrl: String = "https://gist.githubusercontent.com/TerrorJack/a60ff752270c40a6485ee787837390aa/raw/119cbacb29dc12d74ae676b4b02687a8f38b02e4/L2Regularization.sc"
In [10]:
val AdamPluginUrl = "https://gist.githubusercontent.com/Rabenda/0c2fc6ba4cfa536e4788112a94200b50/raw/233cbc83932dad659519c80717d145a3983f57e1/Adam.sc"
interp.load(scala.io.Source.fromURL(new java.net.URL(AdamPluginUrl)).mkString)
Out[10]:
AdamPluginUrl: String = "https://gist.githubusercontent.com/Rabenda/0c2fc6ba4cfa536e4788112a94200b50/raw/233cbc83932dad659519c80717d145a3983f57e1/Adam.sc"
In [12]:
// `interp.load` is a workaround for https://github.com/lihaoyi/Ammonite/issues/649 and https://github.com/scala/bug/issues/10390
interp.load("""
  val hyperparameters = Factory[Builtins with CNNs with L2Regularization  with Adam with INDArrayLearningRate ].
   newInstance(learningRate = 1e-4, l2Regularization = 0.000001)
""")

Write softmax

About softmax classifier, let's see SoftmaxLinearClassifier

In [13]:
import hyperparameters.implicits._
Out[13]:
import hyperparameters.implicits._
In [14]:
import hyperparameters.INDArrayLayer

def softmax(scores: INDArrayLayer): INDArrayLayer = {
  val expScores = hyperparameters.exp(scores)
  (expScores + 1e-8) / (expScores.sum(1) + 1e-8)
}
Out[14]:
import hyperparameters.INDArrayLayer


defined function softmax

Prepare data

Read data

To read the images and corresponding label information for test data from CIFAR10 database and process them, we need Cifar10. This library contains the read and processed CIFAR10 data, provided in this course.

Process data

Before passing data to the model, we need process label data with (one hot encoding): transform INDArray of NumberOfPixels × 1 into INDArray of NumberOfPixels × NumberOfClasses. The value of correct classification corresponding to each line is 1, and the values of other columns are 0. The reason for differentiating the training set and test set is to make it clear that whether the network is over trained which leads to overfitting.

But Cifar10 also process the above content.

In [15]:
import $ivy.`com.thoughtworks.deeplearning.etl::cifar10:1.1.0`
import com.thoughtworks.deeplearning.etl.Cifar10
import com.thoughtworks.future._
val cifar10 = Cifar10.load().blockingAwait
Out[15]:
import $ivy.$                                                 

import com.thoughtworks.deeplearning.etl.Cifar10

import com.thoughtworks.future._

cifar10: com.thoughtworks.deeplearning.etl.Cifar10 = Cifar10(
  Vector(
...

Compose your neural network

Define all layers and initialize Weight

In [16]:
import hyperparameters.INDArrayWeight
Out[16]:
import hyperparameters.INDArrayWeight
In [17]:
val NumberOfClasses: Int = 10
val NumberOfPixels: Int = 3072
val KernelSize: Int = 7
val KernelWidth: Int = KernelSize
val KernelHeight: Int = KernelSize
val NumFilters: Int = 32
val HiddenDim: Int = 500 //define hidden_layer->affineRuleOfCnnLayer
val WeightScale: Double = 1e-2 
def PixelHeight = Cifar10.Height
def PixelWidth = Cifar10.Width
val Padding: Int = (KernelSize - 1) / 2
val Stride: Int = 1
val PoolSize: Int = 2
Out[17]:
NumberOfClasses: Int = 10
NumberOfPixels: Int = 3072
KernelSize: Int = 7
KernelWidth: Int = 7
KernelHeight: Int = 7
NumFilters: Int = 32
HiddenDim: Int = 500
WeightScale: Double = 0.01
defined function PixelHeight
defined function PixelWidth
Padding: Int = 3
Stride: Int = 1
PoolSize: Int = 2
In [18]:
object AllWeightsAndBias {
    import org.nd4s.Implicits._
    val cnnWeight: INDArrayWeight = INDArrayWeight(Nd4j.randn(Array(NumFilters, Cifar10.NumberOfChannels, KernelHeight, KernelWidth)) * WeightScale)
    val cnnBias: INDArrayWeight = INDArrayWeight(Nd4j.zeros(NumFilters))
    val affineWeight: INDArrayWeight = INDArrayWeight(Nd4j.randn(Array(NumFilters * (PixelHeight / PoolSize) * (PixelWidth / PoolSize), HiddenDim)) * WeightScale)
    val affineBias: INDArrayWeight = INDArrayWeight(Nd4j.zeros(HiddenDim))
    val affineLastWeight: INDArrayWeight = INDArrayWeight(Nd4j.randn(Array(HiddenDim, Cifar10.NumberOfClasses)) * WeightScale)
    val affineLastBias: INDArrayWeight = INDArrayWeight(Nd4j.zeros(Cifar10.NumberOfClasses))
}
Out[18]:
defined object AllWeightsAndBias
In [19]:
import AllWeightsAndBias._
Out[19]:
import AllWeightsAndBias._
In [20]:
def affine(input: INDArrayLayer, weight: INDArrayWeight, bias: INDArrayWeight): INDArrayLayer = {
    input dot weight + bias
}

def relu(input: INDArrayLayer): INDArrayLayer = {
    import hyperparameters.max
    max(input, 0.0)
}
Out[20]:
defined function affine
defined function relu
In [21]:
def myNeuralNetwork(input: INDArray):  INDArrayLayer = {
    import hyperparameters.max
    import hyperparameters.maxPool
    import hyperparameters.conv2d
    
    val cnnLayer = maxPool(relu(conv2d(input.reshape(input.shape()(0), Cifar10.NumberOfChannels, PixelHeight, PixelWidth), cnnWeight, cnnBias, (KernelHeight, KernelWidth), (Stride, Stride), (Padding, Padding))), (PoolSize, PoolSize))

    val affineRuleOfCnnLayer = relu(affine(cnnLayer.reshape(input.shape()(0), NumFilters * (PixelHeight / PoolSize) * (PixelWidth / PoolSize)), affineWeight, affineBias))

    val affineOfaffineRuleOfCnnLayer = affine(affineRuleOfCnnLayer.reshape(input.shape()(0), HiddenDim), affineLastWeight, affineLastBias)

    val softmaxValue = softmax(affineOfaffineRuleOfCnnLayer)

    softmaxValue
    
}
Out[21]:
defined function myNeuralNetwork

Create LossFunction

To learn about the prediction result of the neural network, we need to write the loss function lossFunction. We use cross-entropy loss to make comparison between this result and the actual result before return the score. Formula:

In [22]:
import hyperparameters.DoubleLayer

def lossFunction(input: INDArray, expectOutput: INDArray): DoubleLayer = { 
    val probabilities = myNeuralNetwork(input)
    -(hyperparameters.log(probabilities) * expectOutput).mean   
}
Out[22]:
import hyperparameters.DoubleLayer


defined function lossFunction

Train your neural network

To observe the training process of the neural network, we need to output loss; while training the neural network, the loss shall be decreasing.

In [23]:
class Trainer(batchSize: Int, numberOfEpoches: Int = 5) {
    import scalaz.std.anyVal._
    import scalaz.syntax.all._
    @volatile
    private var isShuttingDown: Boolean = false

    private val lossBuffer = scala.collection.mutable.Buffer.empty[Double]
        
    def plotLoss(): Unit = Seq(Scatter(lossBuffer.indices, lossBuffer)).plot(title = "loss by time")
    
    def interrupt(): Unit = isShuttingDown = true

    def startTrain(): Unit = {

        @monadic[Future]
        def trainTask: Future[Unit] = {
            isShuttingDown = false
            var epoch = 0
            
            while (epoch < numberOfEpoches && !isShuttingDown) {
                val cifar10 = Cifar10.load().blockingAwait
                val iterator = cifar10.epoch(batchSize).zipWithIndex
                while (iterator.hasNext && !isShuttingDown) {
                    val (Cifar10.Batch(labels, batch), i) = iterator.next()
                    val loss = lossFunction(batch, labels).train.each
                    lossBuffer += loss
                    hyperparameters.logger.info(s"epoch=$epoch iteration=$i batchSize=$batchSize loss=$loss")
                }
                epoch += 1
            }
            hyperparameters.logger.info("Done")
        }
        trainTask.onComplete { tryUnit: scala.util.Try[Unit] => tryUnit.get }
    }
}
Out[23]:
defined class Trainer
In [24]:
val trainBatchSize = 50

val trainer = new Trainer(batchSize = trainBatchSize, numberOfEpoches = 1)
trainer.startTrain()
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Out[24]:
trainBatchSize: Int = 50
trainer: Trainer = $sess.cmd22Wrapper$Helper$Trainer@efdf81b
In [24]:
// trainer.interrupt()  //This can interrupt train
In [25]:
trainer.plotLoss()

Predict your Neural Network

We use the processed test data to verify the prediction result of the model and compute the accuracy. The accuracy shall be about 40%.

In [26]:
def getAccuracyResult(): String = {
    def findMaxItemIndex(iNDArray: INDArray): INDArray = {
        Nd4j.argMax(iNDArray, 1)
    }
    
    def getAccuracy(score: INDArray, testExpectLabel: INDArray): Double = {
        import org.nd4s.Implicits._
        val scoreIndex = findMaxItemIndex(score)
        val expectResultIndex = findMaxItemIndex(testExpectLabel)
        val accINDArray = scoreIndex.eq(expectResultIndex)
        (accINDArray.sumT / score.shape()(0))
    }
    
    
    val accuracyResultBuffer = scala.collection.mutable.Buffer.empty[Double]
    val iterator = cifar10.testBatches(trainBatchSize)
    while (iterator.hasNext) {
        val Cifar10.Batch(testDatalabels, testDataBatch) = iterator.next()
        val predictResult = Await.result(myNeuralNetwork(testDataBatch).predict.toScalaFuture, Duration.Inf)
        val accuracyResult = getAccuracy(predictResult ,testDatalabels)
        accuracyResultBuffer += accuracyResult
    }

    val accuracy = accuracyResultBuffer.sum / accuracyResultBuffer.length
    
    s"${accuracy * 100.0}%"
}


println(s"The accuracy is ${getAccuracyResult()}")
The accuracy is 44.94000000000001%
Out[26]:
defined function getAccuracyResult

Summary

We have learned the follows in this article:

  • Prepare and process CIFAR10 data
  • Build a three-layers model with a convolutional layer and two full connection layers

Download this tutorial

DeepLearning.scala is an open source deep-learning toolkit in Scala created by our colleagues at ThoughtWorks. We're excited about this project because it uses differentiable functional programming to create and compose neural networks; a developer simply writes code in Scala with static typing.