Deep Learning Methods Review.
Neural networks as a modeling technique, have been around for a few decades. The fundamental unit of a neural network is a node, which is loosely based on the biological brain neuron. The connections between neurons are also modeled on biological brains, as is the way these connections develop by training.
In the mid-1980s and early 1990s, many important architectural advancements were made, still, the amount of time and data needed to get good results slowed adoption. In the early 2000s, computational power reached the level required for mass adoption. Since then, several deep learning algorithms have been widely introduced in various application domains. Today the usage of deep learning has become essential due to accuracy and robustness. Deep learning methods very fast emerged and expanded applications in various scientific and engineering domains. Health informatics, energy, urban informatics, safety, security, hydrological systems modeling, economics, bioinformatics, and computational
mechanics have been among the early application domains of deep learning. Below you can find a list of the most popular deep learning algorithms, along with example applications.
Convolutional neural network (CNN)
CNN is one of the most popular architectures. This technique is very often employed for image processing. CNN contains three types of layers with different convolutional, pooling, and fully connected layers. In each CNN, there are two stages for the training process, the
feed-forward stage, and the back-propagation stage. The widely known CNN applications are ZFNet, GoogLeNet, VGGNet, AlexNet, ResNet. Being known for image processing, the
literature includes other application domains like energy, computational mechanics, electronics systems, remote sensing, etc.
Recurrent neural networks (RNN)
RNN is was designed to recognize sequences and patterns such as speech, handwriting, etc. It benefits cyclic connections in the structure which employ recurrent computations to sequentially process the input data. RNN is basically a standard neural network that has been extended across time by having edges which feed into the next time step instead of into the next layer in the same time step. Each of the previous inputs data are kept in a state vector in hidden units, and these state vectors are utilized to compute the outputs. RNN is a relatively new architecture, that’s is why the applications have plenty of room for research and exploration. The energy, hydrological prediction, expert systems, navigation, and economics are the current applications reported in the literature.
Denoising AutoEncoder (DAE)
DAE has been extended from AE as an asymmetrical neural network for learning features from noisy datasets. DAE consists of three main layers, including input, encoding, and decoding. Stacked Denoising AutoEncoder (SDAE), as an unsupervised algorithm, is generated by the DEA method, which can be employed for nonlinear dimensionality reduction. This method is a type of feed-forward neural network and employs a deep architecture with multiple hidden layers and a pre-training strategy. DEA is slowly starting to be known among researchers as an efficient DL algorithm. DEA has already been used in various application domains with promising results. The energy forecasting, cybersecurity, banking, fraud detection, image classification, and speaker verification are among the current popular applications of DEA.
The deep belief networks (DBNs)
DBNs are employed for high dimensional manifolds learning of data. This method contains multiple layers, including connections between the layers except for connections between units within each layer. DBNs can be considered as a hybrid multi-layered neural network, including directed and undirected connections. DBNs contain restricted Boltzmann machines (RBMs) which are trained in a greedy manner. Each RBM layer communicates with both the previous and subsequent layers. This model is consists of a feed-forward network and several layers of restricted Boltzmann machines or RBM as feature extractors. DBN is one of the most reliable deep learning methods with high accuracy and computational efficiency. Thus, the application domains have been widespread, including exciting applications in a wide range of engineering and scientific problems. Human emotion detection, time series prediction, renewable energy prediction, economic forecasting, and cancer diagnosis have been among the applications.
Long Short-Term Memory (LSTM)
LSTM is an RNN method that benefits feedback connections to be used as a general-purpose computer. This method can of for both sequences and pattern recognition and image processing applications. In general, LSTM contains three central units, including input, output, and forget gates. LSTM can control on deciding when to let the input enter the neuron and to remember what was computed in the previous time step. One of the main strengths of the LSTM is that it decides all these based on the current input itself. LSTM has shown great potential in environmental applications, e.g., geological modeling, hydrological prediction, air quality, and hazard modeling. Due to the generalization abilities of LSTM architecture, it can be suitable for many other applications.
Conclusions
Deep learning methods are fast-evolving. Some of them have advanced to be specialized in a particular application domain. Deep Learning is strongly technique-focused. There are not many explanations for each of the new ideas showing up. Most new ideas came out with experimental results attached to prove that they work.