Expert discussions: deep enlightenment learning success, and why it is not a panacea for all AI problem?
Lei Feng network Note: the author of shanshiguang, PhD, calculation of the Chinese Academy of Sciences researcher, PhD, Deputy Director of the CAs key laboratory of intelligent information processing. Mainly engaged in computer vision, pattern recognition, machine learning, and other related research, especially related to face recognition research. This article was originally titled deep learning in computer vision the success of enlightenment and open discussion of the problem.
Large depth of the data supporting the revival of learning AI landmark progress in the field of is, of course, but that doesn't mean deep learning has the potential to solve all the AI problems.
The "Summary" in recent years, deep learning in image classification, object detection and recognition, and even "composition (Image Captioning)" and many other computer vision tasks are a great success. This paper first discusses the depth of learning success brings us some enlightenment, and then discuss the issues related to open. The only personal opinions, inappropriate reader is correct.
| Deep enlightenment learning success
Deep learning success has brought rapid advances in artificial intelligence-related technology, solves many previously considered intractable problems, but more importantly it is to give us a idea changes, I think it is mainly reflected in the following aspects.
1, optimizing method of change is the key to open the door to deep learning Renaissance
Reviews since 2006 (first year of so-called deep learning) in depth study of ten years of development, we must first note that the optimization method for the continuous improvement of an important role. Need to be aware of is that deep learning is not new technology, but the last century 80 's revival of multilayer neural networks. CV now sought-after model in the field (such as depth DCNN Convolutional neural networks) has basically taken shape in the 80 's of the last century, was not popular for many reasons, one of which was a chronic lack of efficient way to efficiently optimize multiple network, in particular for multilayer neural networks initialize methods effectively.
In this sense, Hinton, 2006 major contribution is to create unsupervised, tiered training multi-layer neural networks pioneer, so that many researchers regained confidence in multilayer neural network.
But for the last three years DCNN's prosperity and unsupervised, tiered training does not make much difference, more optimization method or to improve skills related to such Mini-Batch SGD, ReLU activation function, Batch Normalization, the shortcut in the ResNet etc and, in particular, means dealing with gradients disappear, for DCNN network continues to deepen, performance continues to improve, too.
2 man-made features, experience-driven paradigm to data-driven learning paradigm
Prior to the rise of deep learning, expert knowledge and experience-driven AI paradigm dominated the speech processing, computer vision and pattern recognition in many areas for many years, particularly in terms of information and feature design, past heavy reliance on artificial, seriously affect the effectiveness of intelligent processing technology and versatility. Deep learning upended that "artificial characteristics" paradigm, opening a data-driven "learning" model. Embodied in two points:
1) so-called experience and knowledge in the data, the data volume is large enough without an explicit experience or knowledge is embedded, you can learn directly from the data;
2) can be directly from the original signal started to learn that instead of converting people to a so-called "better" space for learning. Developers of data-driven learning paradigm makes no need to experience and knowledge of process but are not designed for different issues, which greatly improves the versatility of AI algorithms, also significantly reduced the difficulty of solving new problems.
3, from "step by step, divide and conquer" to "learning from end to end"
Partition or process, upcoming or complex problem into a number of simple steps, once a commonly used way to solve complex problems. In the field of AI, and widely used methodology.
For example, in order to solve the image problem of pattern recognition, used to be divided for pre-processing and feature extraction and selection, classifier design of a number of steps. As another example, in order to solve problems, you can use global nonlinear piecewise-linear way. This motive is clear, namely, child or child step easy and manageable, easier to solve. But from the perspective of deep learning, its disadvantages are also equally clear: optimal does not necessarily mean global optimal, each step is the best doesn't mean the whole process is optimal.
In contrast, deep learning emphasizes learning from end to end (end-to-end learning), namely: not to artificial steps or divided into subproblems, but completely to the neural network direct mapping from input to output. Compared to the divide and conquer strategy, end-to-end learning with synergy (synergy) advantage, is more likely to get better on a global solution. Of course, if we have to as a hierarchical "substeps or child problem" is also acceptable, but what those layering completed is not predetermined, but automatically by global optimization based on the data.
4, deep learning we have strong ability of nonlinear modeling
Many complex issues are highly nonlinear in nature and deep learning of nonlinear transformation from input to output, this is the depth of learning in a number of complex issues achieved a breakthrough on one of the important reasons.
Before studying in depth, number of linear or nearly linear models were very popular. Especially since the 90 's of the last century, with linear subspaces of dimension reduction for the purpose of discriminant method to get everyone's attention, such as principal component analysis, Fisher linear discriminant analysis, independent component analysis, and so on.
Later, in order to deal with non-linear problem, Kernel techniques, manifold learning, nonlinear processing methods have been taken seriously. Kernel methods tried to implement a nonlinear transformation of the raw input, but it cannot be defined an explicit nonlinear transformations, with only a limited kind of kernel functions, define the target dot in space, indirectly established.
2000 years after the once-popular attention to the manifold learning methods through the sample between the GEODESIC distance or local neighborhood partnership still to learn about nonlinear mapping, it is regrettable that these methods are difficult to realize non-explicit non-linear transform of training samples. And deep learning by acting on the nonlinear activation function of a large number of neurons (such as the Sigmoid or ReLU), access to a complex nonlinear transformations can be fit enough.
5, models are not always bad Occam's razor principle in many areas, especially in the field of machine learning to be widely known, it warns: "if not necessary, don't add the entity". In other words, the model can be simple to solve the problem better not complex.
This principle in the area of machine learning is important for improving generalization of the model law also makes large complex models tend not to be good. And deep learning is precisely at this point is confusing, in AlexNet, for example, needed to learn the parameters (weight) of up to 60 million, so great arguments seem to indicate that this is a very complex (if not overly complex) model.
Of course, how much needed to learn the parameters in the model does not directly equate to the complexity of the model, but there is little doubt that deep learning seems to be "complexity" is very high. Then Occam's razor principle failure yet? Model complexity or depth of the seemingly complex learning is not high? There seems to be no clear theoretical support. Some recent work suggests that things are already trained, complex, deep learning model can be pruned, compression and other means are reduced, their performance does not diminish or even can be improved.
The key here may lie in "big data" about the "dividend". One possibility is: researchers had long faced a "small data", so too preferred a simple model. And in the sheer amount of data today, moderately complex model to become more responsive to the complex problems faced by researchers, when a large amount of training data and test data distribution and even test data "ran out" when training data, training data on the "fitting" become less scary.
6, neuroscience-inspired idea deserves more attention
Deep learning as a multilayer neural networks are inspired by neuroscience and developed.
Convolutional neural networks in particular, its cognitive roots in 1980 in Fukushima machine model, and the model of motivation is to simulate the receptive field of the mammalian Visual system is getting bigger, layer-by-layer extraction from characteristics of the simple and the complex, enabling semantic chain of abstract visual pathway. The joint efforts of Hubel and Wiesel, the Nobel Prize winner, and the pathway from the 60 's of the last century it has become clear, for CNN was born with good references. But it is worth noting is that biological Visual neural pathways are extremely complex, neuroscientists in the primary visual cortex neurons in a simple edge detection feature is clear, pathways behind some of the more complex the function of nerve cells there to explore, but on a higher level of complicated functions of cells and their mechanism of action is not yet clear.
Depth model such as this means that CNN really simulates the Visual pathway is unclear. But you can be sure of is that the biological connections of the nervous system is very complex, not just bottom-feed and recursive, there is plenty of feedback from top to bottom, as well as external connections from other neural subsystem, which is the current model has not been modeled. Hermes iPhone cover
https://www.youtube.com/watch?v=U414VDsc38o However, advances in brain science can provide more possibilities for model development, is very notable. For example, more recent neuroscience shows that was once considered function to be extremely nerve cells actually has a good plasticity of alienation. For example, a large number of nerve cells in the visual cortex are losing their vision shortly after the demand, which was "reinventing" tactile or other data modes. This plasticity of the nervous system means various intelligent processing task has good versatility, for general development of artificial intelligence to provide a reference.
Large depth of the data supporting the revival of learning AI landmark progress in the field of is, of course, but that doesn't mean deep learning has the potential to solve all the AI problems. Discussion on deep learning of open questions in the field below.
1, by analogy: the necessary data?
Large deep learning is the cornerstone of our success, in depth study of the data, just as fuel in rockets. More and more applications are applications continue to accumulate more wealth of data, which is essential for the further development and application of deep learning. However, excessive reliance on marked one of the biggest limitations of the data is exactly the depth of learning.
Data collection costs, and labeling costs have begun to rise, and there's some there are difficult problems to collect data in the field. For example in the field of medical diagnostics, some of the more rare disease data collection is difficult.
More importantly, when we as a frame of reference when human intelligence, will naturally ask: whether human intelligence is data study results? The answer is not clear. From the individual's perspective, the answer is likely to be negative: we even saw an Apple (or even just an Apple picture) learned to identify Apple without having observed hundreds of different kinds of apples. However, this critical depth appears to be justified, it may not be fair: humans as a species, evolution has seen far more than hundreds or thousands of Apple?
But in any case, a "small data" how to drive deep learning and other machine learning methods is a new direction to explore. In this sense, based on unsupervised learning, similar data transfer in the field of learning, general model in adaptation, knowledge and empirical methods of embedding is very notable.
2, anyway: how to get unsupervised learning ability?
Gets the annotation data of high cost time and money, but not a lot of unsupervised data acquisition cost is negligible. And the current depth of unsupervised learning insufficient data learning capacity, resulting in large amounts of unsupervised data like gold-rich sand sea, we do not have efficient gold rush weapon. Funny thing is, review the history of deep learning, we should remember in 2006, Professor Hinton and others advocated that use deep unsupervised learning for neural network training. But since then, especially after DCNN, unsupervised training seems to have been abandoned by many researchers (especially in the area of CV).
Learn directly from the large amounts of unsupervised data model is very difficult, even if it is one of the "machine", and "Wolf child" example of warning us "self-taught" seems to be unrealistic. But "few instructors data + a lot of unsupervised data" model might be more worthy of study.
3, from learning to structure learning?
Deep learning with "data-driven" paradigm subvert the "artificial characteristics" paradigm, which is a major step forward. But at the same time it yourself into a "man-made structures" by donating money.
Hinton group was originally designed by Professor AlexNet and is then VGG,GoogLeNet,ResNet, and so on, are experts in experience design. Given a new problem, exactly what kind of network is the best (such as convolution) is unknown, which to some extent prevented deep learning in more tasks on the popularity and usage. Therefore, the learning network structure and network parameters is a worthy concern of research directions.
From a computational point of view, the comprehensive learning network structure is extremely complex. Despite some attempts in this regard in the near future, such as pruning algorithms, network reduction, you can adjust the network structure to a certain extent. But also a few parameters to the network (as kernel DCNN bank number) for exploratory work to learn, but is still at an early stage.
4, how to forecast stages for feedback modulation to the network? Hermes iPhone 5 Case
We know that in the realization of the human visual system to "see" in the process, Visual pathways, neurons receive input not only from lower or layers of neurons, also accepts high level feedback signals of neurons, and from other neural subsystem (hearing) the modulation of the signal.
In contrast, we currently used advanced neural network (especially DCNN) in training after the completion of the feature extraction or forecast stages, low-level most neurons cannot accept high neural feedback signal, there is no mechanism to receive other signals (such as prior or other modal information) modulation. This means that a priori knowledge, context, guesses and imagination (head up) "smart" capabilities in the existing application and reflected on the deep Web. How to break through this predicament, given Web-aware adaptive modulation stage, is a worthy study.
5, how to give machine deduction?
Based on data for deep learning can be thought of as a kind of induction, but from the General principles of interpretation is another important human ability, particularly in the cognitive and decision-making processes, we rely heavily on deductive reasoning.
Deductive reasoning in many cases seems to has nothing to do with the data. For example, even if it does not give any examples, we can also rely on symbols (language) to describe, to learn to recognize something that had never seen before.
This problem seems beyond the reach of deep learning zero-shot learning areas, but perhaps not impossible. For example, in recent years an increasing number of production models based on depth of learning are trying to achieve from the symbols (concepts) to image formation.
"The author" shanshiguang, PhD, calculation of the Chinese Academy of Sciences researcher, PhD, Deputy Director of the CAs key laboratory of intelligent information processing. Mainly engaged in computer vision, pattern recognition, machine learning, and other related research, especially related to face recognition research. Has been recommended by the computer society of class a international journals and conferences, published more than 50 papers, papers in Google Scholar references more than 9,000 times. Have been invited to serve as ICCV,ACCV,ICPR,FG International Conference on multiple areas, such as President (Area Chair), the current IEEE Trans. on Image Processing,Neurocomputing and Pattern Recognition Letters Editorial Board of international journals such as (AE). Research by the 2005 National Science and technology progress award second prize of national natural science Award and 2015, 2012 Fund "green" winners, 2015-CCF Youth Science Prize winners.
Lei Feng network Note: this paper consists of deep learning Forum (public) authorized network of Lei Feng (search for "Lei feng's network", public interest) released, please contact the authorized, and be sure to keep the author and source, no deletion of content.