In DeepLearning.scala 2.0, all features are provided by plugins.

A plugin is a Scala trait that can be mixed-in with other plugins. Plugins can either provide new features or modify behaviors of existing features.

Here are some examples of what plugins can do:

  • An optimization algorithm
  • A special subnetwork
  • A primary differentiable operator
  • Logging
  • Integration of a new back-end other than nd4j

How to use plugins?

Typically, plugins of a neural network are usually setup as following:

import com.thoughtworks.feature.Factory
val hyperparameters = Factory[Plugin1 with Plugin2 with Plugin3].newInstance(
  hyperparameterKey1 = hyperparameterValue1,
  hyperparameterKey2 = hyperparameterValue2,
  hyperparameterKey3 = hyperparameterValue3
)
import hyperparameters.implicits._

hyperparameterKey1, hyperparameterKey2 and hyperparameterKey3 are settings required by Plugin1, Plugin2 and Plugin3. Those hyperparameter keys may differ when different plugins are used.

Then, the neural network can be built from functions and types in hyperparameters.

val weight1 = hyperparameters.INDArrayWeight(initialValue1)
val weight2 = hyperparameters.INDArrayWeight(initialValue2)
val weight3 = hyperparameters.INDArrayWeight(initialValue3)
def myNeuralNetwork(input: INDArray): hyperparameters.INDArrayLayer = {
  val layer1 = hyperparameters.function1(input, weight1)
  val layer2 = layer1.method1(weight2)
  hyperparameters.function2(layer2, weight3)
}

INDArrayLayer and INDArrayWeight, function1, function2 and method1 are types and functions provided by plugins. Also, those types and functions may differ when different plugins are used.

Since Scala traits can be nested, the term plugin is only used to refer to the root level trait. For example:

  • Plugin1#INDArrayLayerApi is a mixin trait but not a plugin;
  • Plugin1 is the plugin.

Plugin users only need to create hyperparameters object from plugins (aka root level mixins), then all the nested types of same name in those plugins will get mixed-in automatically.

See Getting Started for a complete example of the usage of plugins.

Built-in plugins

Every DeepLearning.scala 2.0 built-in plugin is a library distributed on Maven central repository. The artifact ID is the lower cased plugin name with a plugins- prefix, and the group ID is com.thoughtworks.deeplearning.

The complete list of all built-in plugins can be found in Scaladoc of com.thoughtworks.deeplearning.plugins package.

In most cases, you may want to enable all built-in plugins at once. Then you only need the Builtins plugin, which contains all other built-in plugins.

To install the Builtins plugin for a sbt project, you can add the following settings in your build.sbt:

libraryDependencies += "com.thoughtworks.deeplearning" %% "plugins-builtins" % "latest.release"

To install the plugin in Jupyter Scala or Ammonite REPL, you can use magic imports.

import $ivy.`com.thoughtworks.deeplearning::plugins-builtins:2.0.0`

Then, in the Scala source file, use the Builtins plugin as following:

import com.thoughtworks.deeplearning.plugins.Builtins
val hyperparameters = Factory[Builtins].newInstance()

Non-built-in plugins

Most of non-built-in plugins are distributed as source files, as simple as a Gist.

Plugin Name Plugin Description
FixedLearningRate Setup fixed learning rate when training INDArray weights.
Adagrad An adaptive gradient algorithm with per-parameter learning rate for INDArray weights.
L1Regularization L1 Regularization.
L2Regularization L2 Regularization.
Momentum The Momentum and NesterovMomentum optimizer for SGD.
RMSprop The RMSprop optimizer for SGD.
Adam The Adam optimizer for SGD.
INDArrayDumping A plugin to dump weight matrices during training.
CNN A standalone Convolutional Neural Network implementation.
Add your own algorithms, models or any cool features here.

See Contributor Guide for the guideline to build a plugin.

DeepLearning.scala is an open source deep-learning toolkit in Scala created by our colleagues at ThoughtWorks. We're excited about this project because it uses differentiable functional programming to create and compose neural networks; a developer simply writes code in Scala with static typing.