Gradient Descent

Float Feedforward Neuron

Constructors

FloatFeedForwardNeuron(_prevs, _num_prevs, _query_manager, &_activ)

• _prevs(FloatUnitNeuron**)
  Array of memory addresses pointing to FloatUnitNeuron and its subclass instances
  Indicates previous neurons that the current neuron derives state data from
  If the assigned array consists of pointers to subclasses of FloatUnitNeuron, include (FloatUnitNeuron**) in front of the variable
• _num_prevs
  Number of elements (int) in the _prevs array
• _query_manager(FeedbackQueryManager*)
  Reference to the FeedbackQueryManager instance
• &_activ(std::string const)
  String literal of either of the following indicating the activation function of the neuron:
  • "identity"
  • "relu"
  • "tanh"
  • "sigmoid"
  For neurons with softmax function as activation function, set the activation function as "identity" and use an external softmax function float* softmax (float* x, int size) located at the header file

FloatFeedForwardNeuron(_prevs, _num_prevs, _query_manager, float (*_activation) (float), float (*_gradient) (float))

• _prevs
  Array of memory addresses pointing to FloatUnitNeuron and its subclass instances
• _num_prevs
  Number of elements (int) in _prevs
• _query_manager
  Reference to the FeedbackQueryManager instance
• float (*_activation) (float)
  Unnamed activation function
  Function pointer that depicts the activation function
• float (*_gradient) (float)
  Unnamed derivative of _activation
  Function pointer that depicts the derivative of _activation
  The input float is synonymous to that of the activation function

Structure

Public Methods and Variables

• lr(floating point value)
  Learning rate for weight update
  The default value is set to 0.7f
• void feedforward()
  Weight values stored in memory are multiplied with state values from previous neurons and put through the activation function defined during construction.
• void feedback(float* fb_input)
  Performs gradient descent operation on the specific neuron and produces instances of FeedbackQuery which are added to the internal array in the assigned query_manager
  Gradients of each neuron are calculated analytically. Have a look at the code for details.
  For more descriptive derivation, look at "Last Layer" and "Hidden Layers" sections of the "Gradient Descent and Back Propagation" article by Tobias Hill in the Medium .

Additional Notes and Code

For more details on private variables, methods, and other functionalities performed within the definition of the class, refer to the hpp file and the cpp file .

Float Gradient Descent

Subclass of GlobalOperator that starts the gradient descent training process of the neural network

Constructors

FloatGradientDescent(_targets, _num_targets)

• _targets
  Array of memory addresses to FloatFeedForwardNeuron used as the output neurons
• _num_targets
  Number of elements (int) in _targets

Training

FloatGradientDescent contains functions that executes the feedback(float* fb_input) of the output neurons.

However, before executing execute(), the global operator must calculate the loss function given a correct output (refer to supervised learning).

Minimum Square Loss

Full list of tools for MSL calculation can be found here .

void calculate_l1_loss(int* indices, int length, float* correct_value, float* coef)

• indices
  Indices of specific neurons to start feedback loop from based on the internal targets array
• length
  Number of elements in indices
• correct_value
  Array of correct value to calculate MSL from
• coef
  Coefficient value to multiply the calculated gradients with

Cross Entropy Loss

Full list of tools for CEL calculation can be found here .

void calculate_cross_entropy_loss(float* correct_value, float* coef)

• correct_value
  Array of correct value to calculate CEL from
• coef
  Coefficient value to multiply the calculated gradients with

void calculate_cross_entropy_loss(int index, float coef)

• index
  The correct index (used in classification problems)
• coef
  Coefficient value to multiply the calculated gradients with

Code

For more details on private variables, methods, and other functionalities performed within the definition of the class, refer to the hpp file and the cpp file .