Extreme Learning Machine for Simple Classification

So last week, my friend in college asked for my help about implementing the code of extreme learning machine. At that time, I didn’t quite understand about the algorithm — but I decided to help anyway while learning and understanding it. Instead of just helping my friend, I want to share my learning experience and help people who have difficulty understanding this algorithm.

DDI Editor's Pick: 5 Machine Learning Books That Turn You from Novice to Expert | Data Driven…

Actually, I want to do this a long time ago; to share what I’ve been learning through writing, but don’t know where to start it. So instead of worrying, THIS IS IT! my first post about simple implementation of extreme learning machine.

Extreme Learning Machine

Extreme Learning Machine is a simple learning algorithm for Single-Layer Feed-Forward Neural Network (SLFN). In theory, the Extreme Learning Machine algorithm (ELM) tends to provide good performance at extremely fast learning speed.

Unlike traditional feedforward network learning algorithms like back-propagation (BP) algorithm, the ELM does not use a gradient-based technique. With this method, all the parameters are tuned once. This algorithm does NOT need iterative training.

ELM Implementation

1. Create the random weights matrix and bias for the input layer.

The size of the weight matrix and bias is (j x k) and (1 x k) where j is the number of hidden nodes and k is the number of input nodes.

2. Calculate the hidden layer output matrix

The initial hidden layer output matrix is calculated by multiplying X which is training data with transpose of weight matrix

3. Choose activation function

You can choose any activation function that you want. But in this example, I will choose sigmoid activation function because it is easy to implement.

4. Calculate the Moore-Penrose pseudoinverse

Several methods can be used to calculate the Moore–Penrose generalized inverse of H. These methods may include but are not limited to orthogonal projection, orthogonalization method, iterative method, and singular value decomposition (SVD).

5. Calculate the output weight matrix beta

6. Repeat step 2 for the testing dataset, creating a new H matrix. After that, create the result matrix called ŷ. We use the already-known beta matrix.

Data

We will make random classification data using make_classification library from sklearn.

Model

Test Model

Accuracy: 0.89

Our model resulted in an accuracy of 0.89. This can be improved by many factors such as:

• The activation function that we use
• Number of the hidden neuron
• If the data is normalized or not

Conclusion

Extreme Learning Machine algorithm is one of the most efficient machine learning algorithms in neural networks. Because of the non-iterative training, all the parameters are tuned once. This results in a high training speed. Its implementation is easy to understand, and it can be used to solve complex problems.

Thanks for reading! we have reached the end of this article, hopefully you found this interesting and fun. If you have any questions and/or suggestions, let me know in the comments.

References:

[1] Extreme learning machine: Theory and applications Guang-Bin Huang, Qin-Yu Zhu, Chee-Kheong Siew.