动态神经网络综述

图神经网络研究综述近年来，图神经网络（graph neural networks，GNNs）成为了机器学习领域的研究热点。

图神经网络是一种用于处理图数据的神经网络模型，通过学习节点之间的关系和图的结构，实现对图数据的分析和预测。

本文将对图神经网络的研究进行综述，并介绍其应用和未来发展方向。

一、图神经网络简介图神经网络是一种基于图结构的神经网络模型，其目标是学习节点之间的关系和图的结构。

相比于传统的神经网络模型，图神经网络可以处理非结构化数据，并在节点分类、图分类、关系预测等任务中取得良好的效果。

图神经网络主要由节点嵌入和图嵌入两个部分组成。

1. 节点嵌入节点嵌入是将节点映射到低维空间中的一个过程，将图中的节点表示为向量形式，用于捕捉节点的特征和上下文信息。

常用的节点嵌入方法有GraphSage、GAT和GCN等。

2. 图嵌入图嵌入是将整个图映射到低维空间中的一个过程，用于对整个图进行特征提取和表示。

常用的图嵌入方法有Graph Convolutional Networks (GCNs)、Graph Attention Networks (GAT)和GraphSAGE等。

二、图神经网络的应用图神经网络在各个领域都有广泛的应用，包括社交网络分析、推荐系统、生物信息学和药物发现等。

以下将介绍几个典型的应用场景：1. 社交网络分析社交网络是一个充满复杂关系的图结构，传统的方法往往无法准确地分析和预测社交网络中的节点行为。

图神经网络通过学习节点之间的关系和图的结构，可以有效地进行社交网络分析，包括社区发现、影响力传播和节点分类等任务。

2. 推荐系统推荐系统需要对用户和商品之间的关系进行建模和预测，而这种关系往往可以用图结构进行表示。

图神经网络可以学习用户和商品之间的关系，提取隐含的特征信息，并用于推荐算法中的推荐和评分预测。

3. 生物信息学生物信息学研究需要对蛋白质、基因和化合物等生物实体进行建模和分析。

综述人工神经网络在地基沉降预测中的应用摘要：人工神经网络在近几年来发展迅速，在岩土工程界得到了广泛的应用，尤其在地基沉降预测方面取得了突出了成绩，本文将结合现有的一些工程实例来简单地综述一下人工神经网络在地基沉降预测方面的优越性。

关键词：人工神经网络地基沉降随着我国经济的发展，高速公路，高层建筑等作为基础建设的一部分，也得到了迅猛地发展。

这些基础建设中最首要的任务就是地基处理，因此对地基沉降预测就成了工程建设者需要解决的首要问题之一。

目前，对地基沉降预测的方法很多，除了传统的计算方法以外，还有可靠度分析法、沉降差法、FLAC有限差分法等。

近几年，随着人工神经网络方法在岩土工程界的应用，利用人工神经网络方法来预测地基的沉降已取得的比较显著的成绩，本文将结合前人的一些工程实例来综述人工神经网络在地基沉降预测中的优越性。

1人工神经网络的简介人工神经网络（Artificial Neural Network，简称ANN）[1]是集多种现代科学技术为一体的一门新兴实用科学技术。

神经网络反映了人脑功能的基本特性，是人脑的抽象、简化，模拟它的信息处理是由神经元之间的相互作用来实现的；知识与信息的存储表现为网络元件互连间分布式的物理联系；学习和识别取决于各神经元连接权值的动态变化过程。

人工神经网络正是在人类对其大脑神经网络认识理解的基础上人工构造的能够实瑰某种功能的神经网络。

它是理论化的人脑神经网络的数学模型，是基于模仿大脑神经网络结构和功能而建立的一种信息处理系统。

它实际上是由大量简单元件相互连接而成的复杂网络，具有高度的非线性，能够进行复杂的逻辑操作和非线性关系实现的系统。

2BP建模的基本思路2.1 BP神经网络原理[2]BP神经网络（Error Back – Propagation，简称EBP或BP神经网络模型）是一种具有三层或三层以上阶层结构的、采用多层前馈神经网络的误差逆传模型。

层间各神经元实现全连接，即下层的每一个单元与上层的每个单元都实现权连接，而每层神经元之间不连接。

文献综述电气工程及自动化BP神经网络研究综述摘要：现代信息化技术的发展，神经网络的应用范围越来越广，尤其基于BP算法的神经网络在预测以及识别方面有很多优势。

本文对前人有关BP神经网络用于识别和预测方面的应用进行归纳和总结，并且提出几点思考方向以作为以后研究此类问题的思路。

关键词：神经网络；数字字母识别；神经网络的脑式智能信息处理特征与能力使其应用领域日益扩大，潜力日趋明显。

作为一种新型智能信息处理系统，其应用贯穿信息的获取、传输、接收与加工各个环节。

具有大家所熟悉的模式识别功能，静态识别例如有手写字的识别等，动态识别有语音识别等，现在市场上这些产品已经有很多。

本文查阅了中国期刊网几年来的相关文献包括相关英文文献，就是对前人在BP神经网络上的应用成果进行分析说明，综述如下：（一）B P神经网络的基本原理BP网络是一种按误差逆向传播算法训练的多层前馈网络它的学习规则是使用最速下降法，通过反向传播来不断调整网络的权值和阀值，使网络的误差平方最小。

BP网络能学习和存贮大量的输入- 输出模式映射关系,而无需事前揭示描述这种映射关系的数学方程.BP神经网络模型拓扑结构包括输入层（input）、隐层(hide layer)和输出层(output layer)，如图上图。

其基本思想是通过调节网络的权值和阈值使网络输出层的误差平方和达到最小，也就是使输出值尽可能接近期望值。

（二）对BP网络算法的应用领域的优势和其它神经网络相比，BP神经网络具有模式顺向传播，误差逆向传播，记忆训练，学习收敛的特点，主要用于：（1）函数逼近：用输入向量和相应的输出向量训练一个网络以逼近一个函数；（2）模式识别：用一个待定的输出向量将它与输入向量联系起来；（3）数据压缩：减少输出向量维数以便于传输或存储；（4）分类：把输入向量所定义的合适方式进行分类；]9[BP网络实质上实现了一个从输入到输出的映射功能，，而数学理论已证明它具有实现任何复杂非线性映射的功能。

人工神经网络历史发展及应用综述1、引言人类为了生存在改造探索自然的过程中，学会利用机械拓展自身的体力，随着对自然认识的不断深入，创造语言，符号，算盘、计算工具等来强化自身脑力。

复杂的数字计算原本是靠人脑来完成的，为了摆脱这种脑力束缚发明了计算机。

其数字计算能力比人脑更强，更快、更准。

计算机的出现，人类开始真正有了一个可以模拟人类思维的工具，期盼可以实现人工智能，构造人脑替代人类完成相应工作。

要模拟人脑的活动，就要研究人脑是如何工作的，要怎样模拟人脑的神经元。

人脑的信息处理具有大规模并行处理、强容错性和自适应能力、善于联想、概括、类比和推广的特点，多少年以来，人们从生物学、医学、生理学、哲学、信息学、计算机科学、认知学、组织协同学等各个角度企图获悉人脑的工作奥秘，寻求神经元的模拟方法。

在寻找上述问题答案的研究过程中，从20世纪40年代开始逐渐形成了一个新兴的边缘性交叉学科，称之为“神经网络”，是人工智能、认知科学、神经生理学、非线性动力学、信息科学、和数理科学的“热点”。

关于神经网络的研究包含众多学科领域，涉及数学、计算机、人工智能、微电子学、自动化、生物学、生理学、解剖学、认知科学等学科，这些领域彼此结合、渗透，相互推动神经网络研究和应用的发展。

2、定义思维学普遍认为，人类大脑的思维有三种基本方式，分为抽象（逻辑）思维、形象（直观）思维和灵感（顿悟）思维。

逻辑性的思维是根据逻辑规则进行推理的过程，这一过程可以写成指令，让计算机执行，获得结果。

而直观性（形象）的思维是将分布式存储的信息综合起来，结果是忽然间产生想法或解决问题的办法。

这种思维方式的有以下两个特点：一是信息通过神经元上的兴奋模式分布储在网络上；二是信息处理通过神经元之间同时相互作用的动态过程来完成的。

人工神经网络就是模拟第二种人类思维方式。

人工神经网络是由大量具备简单功能的人工神经元相互联接而成的自适应非线性动态系统。

虽然单个神经元的结构和功能比较简单，但大量神经元连接构成的网络系统行为却异常复杂。

深度神经网络的优化方法综述随着大数据的普及和应用场景的不断拓展，深度神经网络(DNN)成为现代机器学习和人工智能的核心技术之一。

然而，由于深度神经网络的模型复杂度、参数量、计算量、训练难度等因素，使得其优化方法成为该领域的研究热点和难点之一。

本文将针对深度神经网络的优化方法进行综述和总结，从梯度下降、优化策略、正则化等方面展开阐述。

1. 梯度下降算法梯度下降算法是深度神经网络中最基础、也是最常用的优化方法之一。

其核心思想是通过计算目标函数对于参数的一阶导数，找到能够使目标函数值下降最快的方向，并沿着该方向进行参数更新。

虽然梯度下降算法简单易懂，但其容易陷入局部极小值，收敛速度较慢的缺陷限制了其在深度神经网络优化中的应用。

为了解决梯度下降算法的缺陷，学者们提出了一系列的改进算法，如随机梯度下降(Stochastic Gradient Descent, SGD)、批量梯度下降(Batch Gradient Descent, BGD)、Adam和Adagrad等。

2. 优化策略优化策略是深度神经网络优化的关键，它与学习率、动量、权重衰减等因素密切相关。

当前研究中，主要的优化策略包括自适应学习率、随机停止、动量算法、启发式算法和克服退化策略等。

自适应学习率是指根据当前梯度状态动态调整学习率。

Adagrad算法是一种基于自适应学习率的优化方法，它可以根据梯度值所在区间对学习率进行调整，有效地解决了梯度稀疏问题。

随机停止是将深度神经网络优化过程视为统计学问题，当目标函数和梯度的随机变化达到一定水平后，即停止优化过程。

该策略通常用于对大规模数据集的训练，以避免深度神经网络的过拟合。

动量算法是基于牛顿力学的动量来模拟优化过程。

通过增加历史梯度信息的惯性度量，可以使梯度下降的更新方向在一定程度上保持稳定，在梯度较小的区间实现较快的收敛速度。

其中，Momentum和Nesterov Accelerated Gradient(NAG)是较常用的代表性算法。

Draft:Deep Learning in Neural Networks:An OverviewTechnical Report IDSIA-03-14/arXiv:1404.7828(v1.5)[cs.NE]J¨u rgen SchmidhuberThe Swiss AI Lab IDSIAIstituto Dalle Molle di Studi sull’Intelligenza ArtificialeUniversity of Lugano&SUPSIGalleria2,6928Manno-LuganoSwitzerland15May2014AbstractIn recent years,deep artificial neural networks(including recurrent ones)have won numerous con-tests in pattern recognition and machine learning.This historical survey compactly summarises relevantwork,much of it from the previous millennium.Shallow and deep learners are distinguished by thedepth of their credit assignment paths,which are chains of possibly learnable,causal links between ac-tions and effects.I review deep supervised learning(also recapitulating the history of backpropagation),unsupervised learning,reinforcement learning&evolutionary computation,and indirect search for shortprograms encoding deep and large networks.PDF of earlier draft(v1):http://www.idsia.ch/∼juergen/DeepLearning30April2014.pdfLATEX source:http://www.idsia.ch/∼juergen/DeepLearning30April2014.texComplete BIBTEXfile:http://www.idsia.ch/∼juergen/bib.bibPrefaceThis is the draft of an invited Deep Learning(DL)overview.One of its goals is to assign credit to those who contributed to the present state of the art.I acknowledge the limitations of attempting to achieve this goal.The DL research community itself may be viewed as a continually evolving,deep network of scientists who have influenced each other in complex ways.Starting from recent DL results,I tried to trace back the origins of relevant ideas through the past half century and beyond,sometimes using“local search”to follow citations of citations backwards in time.Since not all DL publications properly acknowledge earlier relevant work,additional global search strategies were employed,aided by consulting numerous neural network experts.As a result,the present draft mostly consists of references(about800entries so far).Nevertheless,through an expert selection bias I may have missed important work.A related bias was surely introduced by my special familiarity with the work of my own DL research group in the past quarter-century.For these reasons,the present draft should be viewed as merely a snapshot of an ongoing credit assignment process.To help improve it,please do not hesitate to send corrections and suggestions to juergen@idsia.ch.Contents1Introduction to Deep Learning(DL)in Neural Networks(NNs)3 2Event-Oriented Notation for Activation Spreading in FNNs/RNNs3 3Depth of Credit Assignment Paths(CAPs)and of Problems4 4Recurring Themes of Deep Learning54.1Dynamic Programming(DP)for DL (5)4.2Unsupervised Learning(UL)Facilitating Supervised Learning(SL)and RL (6)4.3Occam’s Razor:Compression and Minimum Description Length(MDL) (6)4.4Learning Hierarchical Representations Through Deep SL,UL,RL (6)4.5Fast Graphics Processing Units(GPUs)for DL in NNs (6)5Supervised NNs,Some Helped by Unsupervised NNs75.11940s and Earlier (7)5.2Around1960:More Neurobiological Inspiration for DL (7)5.31965:Deep Networks Based on the Group Method of Data Handling(GMDH) (8)5.41979:Convolution+Weight Replication+Winner-Take-All(WTA) (8)5.51960-1981and Beyond:Development of Backpropagation(BP)for NNs (8)5.5.1BP for Weight-Sharing Feedforward NNs(FNNs)and Recurrent NNs(RNNs)..95.6Late1980s-2000:Numerous Improvements of NNs (9)5.6.1Ideas for Dealing with Long Time Lags and Deep CAPs (10)5.6.2Better BP Through Advanced Gradient Descent (10)5.6.3Discovering Low-Complexity,Problem-Solving NNs (11)5.6.4Potential Benefits of UL for SL (11)5.71987:UL Through Autoencoder(AE)Hierarchies (12)5.81989:BP for Convolutional NNs(CNNs) (13)5.91991:Fundamental Deep Learning Problem of Gradient Descent (13)5.101991:UL-Based History Compression Through a Deep Hierarchy of RNNs (14)5.111992:Max-Pooling(MP):Towards MPCNNs (14)5.121994:Contest-Winning Not So Deep NNs (15)5.131995:Supervised Recurrent Very Deep Learner(LSTM RNN) (15)5.142003:More Contest-Winning/Record-Setting,Often Not So Deep NNs (16)5.152006/7:Deep Belief Networks(DBNs)&AE Stacks Fine-Tuned by BP (17)5.162006/7:Improved CNNs/GPU-CNNs/BP-Trained MPCNNs (17)5.172009:First Official Competitions Won by RNNs,and with MPCNNs (18)5.182010:Plain Backprop(+Distortions)on GPU Yields Excellent Results (18)5.192011:MPCNNs on GPU Achieve Superhuman Vision Performance (18)5.202011:Hessian-Free Optimization for RNNs (19)5.212012:First Contests Won on ImageNet&Object Detection&Segmentation (19)5.222013-:More Contests and Benchmark Records (20)5.22.1Currently Successful Supervised Techniques:LSTM RNNs/GPU-MPCNNs (21)5.23Recent Tricks for Improving SL Deep NNs(Compare Sec.5.6.2,5.6.3) (21)5.24Consequences for Neuroscience (22)5.25DL with Spiking Neurons? (22)6DL in FNNs and RNNs for Reinforcement Learning(RL)236.1RL Through NN World Models Yields RNNs With Deep CAPs (23)6.2Deep FNNs for Traditional RL and Markov Decision Processes(MDPs) (24)6.3Deep RL RNNs for Partially Observable MDPs(POMDPs) (24)6.4RL Facilitated by Deep UL in FNNs and RNNs (25)6.5Deep Hierarchical RL(HRL)and Subgoal Learning with FNNs and RNNs (25)6.6Deep RL by Direct NN Search/Policy Gradients/Evolution (25)6.7Deep RL by Indirect Policy Search/Compressed NN Search (26)6.8Universal RL (27)7Conclusion271Introduction to Deep Learning(DL)in Neural Networks(NNs) Which modifiable components of a learning system are responsible for its success or failure?What changes to them improve performance?This has been called the fundamental credit assignment problem(Minsky, 1963).There are general credit assignment methods for universal problem solvers that are time-optimal in various theoretical senses(Sec.6.8).The present survey,however,will focus on the narrower,but now commercially important,subfield of Deep Learning(DL)in Artificial Neural Networks(NNs).We are interested in accurate credit assignment across possibly many,often nonlinear,computational stages of NNs.Shallow NN-like models have been around for many decades if not centuries(Sec.5.1).Models with several successive nonlinear layers of neurons date back at least to the1960s(Sec.5.3)and1970s(Sec.5.5). An efficient gradient descent method for teacher-based Supervised Learning(SL)in discrete,differentiable networks of arbitrary depth called backpropagation(BP)was developed in the1960s and1970s,and ap-plied to NNs in1981(Sec.5.5).BP-based training of deep NNs with many layers,however,had been found to be difficult in practice by the late1980s(Sec.5.6),and had become an explicit research subject by the early1990s(Sec.5.9).DL became practically feasible to some extent through the help of Unsupervised Learning(UL)(e.g.,Sec.5.10,5.15).The1990s and2000s also saw many improvements of purely super-vised DL(Sec.5).In the new millennium,deep NNs havefinally attracted wide-spread attention,mainly by outperforming alternative machine learning methods such as kernel machines(Vapnik,1995;Sch¨o lkopf et al.,1998)in numerous important applications.In fact,supervised deep NNs have won numerous of-ficial international pattern recognition competitions(e.g.,Sec.5.17,5.19,5.21,5.22),achieving thefirst superhuman visual pattern recognition results in limited domains(Sec.5.19).Deep NNs also have become relevant for the more generalfield of Reinforcement Learning(RL)where there is no supervising teacher (Sec.6).Both feedforward(acyclic)NNs(FNNs)and recurrent(cyclic)NNs(RNNs)have won contests(Sec.5.12,5.14,5.17,5.19,5.21,5.22).In a sense,RNNs are the deepest of all NNs(Sec.3)—they are general computers more powerful than FNNs,and can in principle create and process memories of ar-bitrary sequences of input patterns(e.g.,Siegelmann and Sontag,1991;Schmidhuber,1990a).Unlike traditional methods for automatic sequential program synthesis(e.g.,Waldinger and Lee,1969;Balzer, 1985;Soloway,1986;Deville and Lau,1994),RNNs can learn programs that mix sequential and parallel information processing in a natural and efficient way,exploiting the massive parallelism viewed as crucial for sustaining the rapid decline of computation cost observed over the past75years.The rest of this paper is structured as follows.Sec.2introduces a compact,event-oriented notation that is simple yet general enough to accommodate both FNNs and RNNs.Sec.3introduces the concept of Credit Assignment Paths(CAPs)to measure whether learning in a given NN application is of the deep or shallow type.Sec.4lists recurring themes of DL in SL,UL,and RL.Sec.5focuses on SL and UL,and on how UL can facilitate SL,although pure SL has become dominant in recent competitions(Sec.5.17-5.22). Sec.5is arranged in a historical timeline format with subsections on important inspirations and technical contributions.Sec.6on deep RL discusses traditional Dynamic Programming(DP)-based RL combined with gradient-based search techniques for SL or UL in deep NNs,as well as general methods for direct and indirect search in the weight space of deep FNNs and RNNs,including successful policy gradient and evolutionary methods.2Event-Oriented Notation for Activation Spreading in FNNs/RNNs Throughout this paper,let i,j,k,t,p,q,r denote positive integer variables assuming ranges implicit in the given contexts.Let n,m,T denote positive integer constants.An NN’s topology may change over time(e.g.,Fahlman,1991;Ring,1991;Weng et al.,1992;Fritzke, 1994).At any given moment,it can be described as afinite subset of units(or nodes or neurons)N= {u1,u2,...,}and afinite set H⊆N×N of directed edges or connections between nodes.FNNs are acyclic graphs,RNNs cyclic.Thefirst(input)layer is the set of input units,a subset of N.In FNNs,the k-th layer(k>1)is the set of all nodes u∈N such that there is an edge path of length k−1(but no longer path)between some input unit and u.There may be shortcut connections between distant layers.The NN’s behavior or program is determined by a set of real-valued,possibly modifiable,parameters or weights w i(i=1,...,n).We now focus on a singlefinite episode or epoch of information processing and activation spreading,without learning through weight changes.The following slightly unconventional notation is designed to compactly describe what is happening during the runtime of the system.During an episode,there is a partially causal sequence x t(t=1,...,T)of real values that I call events.Each x t is either an input set by the environment,or the activation of a unit that may directly depend on other x k(k<t)through a current NN topology-dependent set in t of indices k representing incoming causal connections or links.Let the function v encode topology information and map such event index pairs(k,t)to weight indices.For example,in the non-input case we may have x t=f t(net t)with real-valued net t= k∈in t x k w v(k,t)(additive case)or net t= k∈in t x k w v(k,t)(multiplicative case), where f t is a typically nonlinear real-valued activation function such as tanh.In many recent competition-winning NNs(Sec.5.19,5.21,5.22)there also are events of the type x t=max k∈int (x k);some networktypes may also use complex polynomial activation functions(Sec.5.3).x t may directly affect certain x k(k>t)through outgoing connections or links represented through a current set out t of indices k with t∈in k.Some non-input events are called output events.Note that many of the x t may refer to different,time-varying activations of the same unit in sequence-processing RNNs(e.g.,Williams,1989,“unfolding in time”),or also in FNNs sequentially exposed to time-varying input patterns of a large training set encoded as input events.During an episode,the same weight may get reused over and over again in topology-dependent ways,e.g.,in RNNs,or in convolutional NNs(Sec.5.4,5.8).I call this weight sharing across space and/or time.Weight sharing may greatly reduce the NN’s descriptive complexity,which is the number of bits of information required to describe the NN (Sec.4.3).In Supervised Learning(SL),certain NN output events x t may be associated with teacher-given,real-valued labels or targets d t yielding errors e t,e.g.,e t=1/2(x t−d t)2.A typical goal of supervised NN training is tofind weights that yield episodes with small total error E,the sum of all such e t.The hope is that the NN will generalize well in later episodes,causing only small errors on previously unseen sequences of input events.Many alternative error functions for SL and UL are possible.SL assumes that input events are independent of earlier output events(which may affect the environ-ment through actions causing subsequent perceptions).This assumption does not hold in the broaderfields of Sequential Decision Making and Reinforcement Learning(RL)(Kaelbling et al.,1996;Sutton and Barto, 1998;Hutter,2005)(Sec.6).In RL,some of the input events may encode real-valued reward signals given by the environment,and a typical goal is tofind weights that yield episodes with a high sum of reward signals,through sequences of appropriate output actions.Sec.5.5will use the notation above to compactly describe a central algorithm of DL,namely,back-propagation(BP)for supervised weight-sharing FNNs and RNNs.(FNNs may be viewed as RNNs with certainfixed zero weights.)Sec.6will address the more general RL case.3Depth of Credit Assignment Paths(CAPs)and of ProblemsTo measure whether credit assignment in a given NN application is of the deep or shallow type,I introduce the concept of Credit Assignment Paths or CAPs,which are chains of possibly causal links between events.Let usfirst focus on SL.Consider two events x p and x q(1≤p<q≤T).Depending on the appli-cation,they may have a Potential Direct Causal Connection(PDCC)expressed by the Boolean predicate pdcc(p,q),which is true if and only if p∈in q.Then the2-element list(p,q)is defined to be a CAP from p to q(a minimal one).A learning algorithm may be allowed to change w v(p,q)to improve performance in future episodes.More general,possibly indirect,Potential Causal Connections(PCC)are expressed by the recursively defined Boolean predicate pcc(p,q),which in the SL case is true only if pdcc(p,q),or if pcc(p,k)for some k and pdcc(k,q).In the latter case,appending q to any CAP from p to k yields a CAP from p to q(this is a recursive definition,too).The set of such CAPs may be large but isfinite.Note that the same weight may affect many different PDCCs between successive events listed by a given CAP,e.g.,in the case of RNNs, or weight-sharing FNNs.Suppose a CAP has the form(...,k,t,...,q),where k and t(possibly t=q)are thefirst successive elements with modifiable w v(k,t).Then the length of the suffix list(t,...,q)is called the CAP’s depth (which is0if there are no modifiable links at all).This depth limits how far backwards credit assignment can move down the causal chain tofind a modifiable weight.1Suppose an episode and its event sequence x1,...,x T satisfy a computable criterion used to decide whether a given problem has been solved(e.g.,total error E below some threshold).Then the set of used weights is called a solution to the problem,and the depth of the deepest CAP within the sequence is called the solution’s depth.There may be other solutions(yielding different event sequences)with different depths.Given somefixed NN topology,the smallest depth of any solution is called the problem’s depth.Sometimes we also speak of the depth of an architecture:SL FNNs withfixed topology imply a problem-independent maximal problem depth bounded by the number of non-input layers.Certain SL RNNs withfixed weights for all connections except those to output units(Jaeger,2001;Maass et al.,2002; Jaeger,2004;Schrauwen et al.,2007)have a maximal problem depth of1,because only thefinal links in the corresponding CAPs are modifiable.In general,however,RNNs may learn to solve problems of potentially unlimited depth.Note that the definitions above are solely based on the depths of causal chains,and agnostic of the temporal distance between events.For example,shallow FNNs perceiving large“time windows”of in-put events may correctly classify long input sequences through appropriate output events,and thus solve shallow problems involving long time lags between relevant events.At which problem depth does Shallow Learning end,and Deep Learning begin?Discussions with DL experts have not yet yielded a conclusive response to this question.Instead of committing myself to a precise answer,let me just define for the purposes of this overview:problems of depth>10require Very Deep Learning.The difficulty of a problem may have little to do with its depth.Some NNs can quickly learn to solve certain deep problems,e.g.,through random weight guessing(Sec.5.9)or other types of direct search (Sec.6.6)or indirect search(Sec.6.7)in weight space,or through training an NNfirst on shallow problems whose solutions may then generalize to deep problems,or through collapsing sequences of(non)linear operations into a single(non)linear operation—but see an analysis of non-trivial aspects of deep linear networks(Baldi and Hornik,1994,Section B).In general,however,finding an NN that precisely models a given training set is an NP-complete problem(Judd,1990;Blum and Rivest,1992),also in the case of deep NNs(S´ıma,1994;de Souto et al.,1999;Windisch,2005);compare a survey of negative results(S´ıma, 2002,Section1).Above we have focused on SL.In the more general case of RL in unknown environments,pcc(p,q) is also true if x p is an output event and x q any later input event—any action may affect the environment and thus any later perception.(In the real world,the environment may even influence non-input events computed on a physical hardware entangled with the entire universe,but this is ignored here.)It is possible to model and replace such unmodifiable environmental PCCs through a part of the NN that has already learned to predict(through some of its units)input events(including reward signals)from former input events and actions(Sec.6.1).Its weights are frozen,but can help to assign credit to other,still modifiable weights used to compute actions(Sec.6.1).This approach may lead to very deep CAPs though.Some DL research is about automatically rephrasing problems such that their depth is reduced(Sec.4). In particular,sometimes UL is used to make SL problems less deep,e.g.,Sec.5.10.Often Dynamic Programming(Sec.4.1)is used to facilitate certain traditional RL problems,e.g.,Sec.6.2.Sec.5focuses on CAPs for SL,Sec.6on the more complex case of RL.4Recurring Themes of Deep Learning4.1Dynamic Programming(DP)for DLOne recurring theme of DL is Dynamic Programming(DP)(Bellman,1957),which can help to facili-tate credit assignment under certain assumptions.For example,in SL NNs,backpropagation itself can 1An alternative would be to count only modifiable links when measuring depth.In many typical NN applications this would not make a difference,but in some it would,e.g.,Sec.6.1.be viewed as a DP-derived method(Sec.5.5).In traditional RL based on strong Markovian assumptions, DP-derived methods can help to greatly reduce problem depth(Sec.6.2).DP algorithms are also essen-tial for systems that combine concepts of NNs and graphical models,such as Hidden Markov Models (HMMs)(Stratonovich,1960;Baum and Petrie,1966)and Expectation Maximization(EM)(Dempster et al.,1977),e.g.,(Bottou,1991;Bengio,1991;Bourlard and Morgan,1994;Baldi and Chauvin,1996; Jordan and Sejnowski,2001;Bishop,2006;Poon and Domingos,2011;Dahl et al.,2012;Hinton et al., 2012a).4.2Unsupervised Learning(UL)Facilitating Supervised Learning(SL)and RL Another recurring theme is how UL can facilitate both SL(Sec.5)and RL(Sec.6).UL(Sec.5.6.4) is normally used to encode raw incoming data such as video or speech streams in a form that is more convenient for subsequent goal-directed learning.In particular,codes that describe the original data in a less redundant or more compact way can be fed into SL(Sec.5.10,5.15)or RL machines(Sec.6.4),whose search spaces may thus become smaller(and whose CAPs shallower)than those necessary for dealing with the raw data.UL is closely connected to the topics of regularization and compression(Sec.4.3,5.6.3). 4.3Occam’s Razor:Compression and Minimum Description Length(MDL) Occam’s razor favors simple solutions over complex ones.Given some programming language,the prin-ciple of Minimum Description Length(MDL)can be used to measure the complexity of a solution candi-date by the length of the shortest program that computes it(e.g.,Solomonoff,1964;Kolmogorov,1965b; Chaitin,1966;Wallace and Boulton,1968;Levin,1973a;Rissanen,1986;Blumer et al.,1987;Li and Vit´a nyi,1997;Gr¨u nwald et al.,2005).Some methods explicitly take into account program runtime(Al-lender,1992;Watanabe,1992;Schmidhuber,2002,1995);many consider only programs with constant runtime,written in non-universal programming languages(e.g.,Rissanen,1986;Hinton and van Camp, 1993).In the NN case,the MDL principle suggests that low NN weight complexity corresponds to high NN probability in the Bayesian view(e.g.,MacKay,1992;Buntine and Weigend,1991;De Freitas,2003), and to high generalization performance(e.g.,Baum and Haussler,1989),without overfitting the training data.Many methods have been proposed for regularizing NNs,that is,searching for solution-computing, low-complexity SL NNs(Sec.5.6.3)and RL NNs(Sec.6.7).This is closely related to certain UL methods (Sec.4.2,5.6.4).4.4Learning Hierarchical Representations Through Deep SL,UL,RLMany methods of Good Old-Fashioned Artificial Intelligence(GOFAI)(Nilsson,1980)as well as more recent approaches to AI(Russell et al.,1995)and Machine Learning(Mitchell,1997)learn hierarchies of more and more abstract data representations.For example,certain methods of syntactic pattern recog-nition(Fu,1977)such as grammar induction discover hierarchies of formal rules to model observations. The partially(un)supervised Automated Mathematician/EURISKO(Lenat,1983;Lenat and Brown,1984) continually learns concepts by combining previously learnt concepts.Such hierarchical representation learning(Ring,1994;Bengio et al.,2013;Deng and Yu,2014)is also a recurring theme of DL NNs for SL (Sec.5),UL-aided SL(Sec.5.7,5.10,5.15),and hierarchical RL(Sec.6.5).Often,abstract hierarchical representations are natural by-products of data compression(Sec.4.3),e.g.,Sec.5.10.4.5Fast Graphics Processing Units(GPUs)for DL in NNsWhile the previous millennium saw several attempts at creating fast NN-specific hardware(e.g.,Jackel et al.,1990;Faggin,1992;Ramacher et al.,1993;Widrow et al.,1994;Heemskerk,1995;Korkin et al., 1997;Urlbe,1999),and at exploiting standard hardware(e.g.,Anguita et al.,1994;Muller et al.,1995; Anguita and Gomes,1996),the new millennium brought a DL breakthrough in form of cheap,multi-processor graphics cards or GPUs.GPUs are widely used for video games,a huge and competitive market that has driven down hardware prices.GPUs excel at fast matrix and vector multiplications required not only for convincing virtual realities but also for NN training,where they can speed up learning by a factorof50and more.Some of the GPU-based FNN implementations(Sec.5.16-5.19)have greatly contributed to recent successes in contests for pattern recognition(Sec.5.19-5.22),image segmentation(Sec.5.21), and object detection(Sec.5.21-5.22).5Supervised NNs,Some Helped by Unsupervised NNsThe main focus of current practical applications is on Supervised Learning(SL),which has dominated re-cent pattern recognition contests(Sec.5.17-5.22).Several methods,however,use additional Unsupervised Learning(UL)to facilitate SL(Sec.5.7,5.10,5.15).It does make sense to treat SL and UL in the same section:often gradient-based methods,such as BP(Sec.5.5.1),are used to optimize objective functions of both UL and SL,and the boundary between SL and UL may blur,for example,when it comes to time series prediction and sequence classification,e.g.,Sec.5.10,5.12.A historical timeline format will help to arrange subsections on important inspirations and techni-cal contributions(although such a subsection may span a time interval of many years).Sec.5.1briefly mentions early,shallow NN models since the1940s,Sec.5.2additional early neurobiological inspiration relevant for modern Deep Learning(DL).Sec.5.3is about GMDH networks(since1965),perhaps thefirst (feedforward)DL systems.Sec.5.4is about the relatively deep Neocognitron NN(1979)which is similar to certain modern deep FNN architectures,as it combines convolutional NNs(CNNs),weight pattern repli-cation,and winner-take-all(WTA)mechanisms.Sec.5.5uses the notation of Sec.2to compactly describe a central algorithm of DL,namely,backpropagation(BP)for supervised weight-sharing FNNs and RNNs. It also summarizes the history of BP1960-1981and beyond.Sec.5.6describes problems encountered in the late1980s with BP for deep NNs,and mentions several ideas from the previous millennium to overcome them.Sec.5.7discusses afirst hierarchical stack of coupled UL-based Autoencoders(AEs)—this concept resurfaced in the new millennium(Sec.5.15).Sec.5.8is about applying BP to CNNs,which is important for today’s DL applications.Sec.5.9explains BP’s Fundamental DL Problem(of vanishing/exploding gradients)discovered in1991.Sec.5.10explains how a deep RNN stack of1991(the History Compressor) pre-trained by UL helped to solve previously unlearnable DL benchmarks requiring Credit Assignment Paths(CAPs,Sec.3)of depth1000and more.Sec.5.11discusses a particular WTA method called Max-Pooling(MP)important in today’s DL FNNs.Sec.5.12mentions afirst important contest won by SL NNs in1994.Sec.5.13describes a purely supervised DL RNN(Long Short-Term Memory,LSTM)for problems of depth1000and more.Sec.5.14mentions an early contest of2003won by an ensemble of shallow NNs, as well as good pattern recognition results with CNNs and LSTM RNNs(2003).Sec.5.15is mostly about Deep Belief Networks(DBNs,2006)and related stacks of Autoencoders(AEs,Sec.5.7)pre-trained by UL to facilitate BP-based SL.Sec.5.16mentions thefirst BP-trained MPCNNs(2007)and GPU-CNNs(2006). Sec.5.17-5.22focus on official competitions with secret test sets won by(mostly purely supervised)DL NNs since2009,in sequence recognition,image classification,image segmentation,and object detection. Many RNN results depended on LSTM(Sec.5.13);many FNN results depended on GPU-based FNN code developed since2004(Sec.5.16,5.17,5.18,5.19),in particular,GPU-MPCNNs(Sec.5.19).5.11940s and EarlierNN research started in the1940s(e.g.,McCulloch and Pitts,1943;Hebb,1949);compare also later work on learning NNs(Rosenblatt,1958,1962;Widrow and Hoff,1962;Grossberg,1969;Kohonen,1972; von der Malsburg,1973;Narendra and Thathatchar,1974;Willshaw and von der Malsburg,1976;Palm, 1980;Hopfield,1982).In a sense NNs have been around even longer,since early supervised NNs were essentially variants of linear regression methods going back at least to the early1800s(e.g.,Legendre, 1805;Gauss,1809,1821).Early NNs had a maximal CAP depth of1(Sec.3).5.2Around1960:More Neurobiological Inspiration for DLSimple cells and complex cells were found in the cat’s visual cortex(e.g.,Hubel and Wiesel,1962;Wiesel and Hubel,1959).These cellsfire in response to certain properties of visual sensory inputs,such as theorientation of plex cells exhibit more spatial invariance than simple cells.This inspired later deep NN architectures(Sec.5.4)used in certain modern award-winning Deep Learners(Sec.5.19-5.22).5.31965:Deep Networks Based on the Group Method of Data Handling(GMDH) Networks trained by the Group Method of Data Handling(GMDH)(Ivakhnenko and Lapa,1965; Ivakhnenko et al.,1967;Ivakhnenko,1968,1971)were perhaps thefirst DL systems of the Feedforward Multilayer Perceptron type.The units of GMDH nets may have polynomial activation functions imple-menting Kolmogorov-Gabor polynomials(more general than traditional NN activation functions).Given a training set,layers are incrementally grown and trained by regression analysis,then pruned with the help of a separate validation set(using today’s terminology),where Decision Regularisation is used to weed out superfluous units.The numbers of layers and units per layer can be learned in problem-dependent fashion. This is a good example of hierarchical representation learning(Sec.4.4).There have been numerous ap-plications of GMDH-style networks,e.g.(Ikeda et al.,1976;Farlow,1984;Madala and Ivakhnenko,1994; Ivakhnenko,1995;Kondo,1998;Kord´ık et al.,2003;Witczak et al.,2006;Kondo and Ueno,2008).5.41979:Convolution+Weight Replication+Winner-Take-All(WTA)Apart from deep GMDH networks(Sec.5.3),the Neocognitron(Fukushima,1979,1980,2013a)was per-haps thefirst artificial NN that deserved the attribute deep,and thefirst to incorporate the neurophysiolog-ical insights of Sec.5.2.It introduced convolutional NNs(today often called CNNs or convnets),where the(typically rectangular)receptivefield of a convolutional unit with given weight vector is shifted step by step across a2-dimensional array of input values,such as the pixels of an image.The resulting2D array of subsequent activation events of this unit can then provide inputs to higher-level units,and so on.Due to massive weight replication(Sec.2),relatively few parameters may be necessary to describe the behavior of such a convolutional layer.Competition layers have WTA subsets whose maximally active units are the only ones to adopt non-zero activation values.They essentially“down-sample”the competition layer’s input.This helps to create units whose responses are insensitive to small image shifts(compare Sec.5.2).The Neocognitron is very similar to the architecture of modern,contest-winning,purely super-vised,feedforward,gradient-based Deep Learners with alternating convolutional and competition lay-ers(e.g.,Sec.5.19-5.22).Fukushima,however,did not set the weights by supervised backpropagation (Sec.5.5,5.8),but by local un supervised learning rules(e.g.,Fukushima,2013b),or by pre-wiring.In that sense he did not care for the DL problem(Sec.5.9),although his architecture was comparatively deep indeed.He also used Spatial Averaging(Fukushima,1980,2011)instead of Max-Pooling(MP,Sec.5.11), currently a particularly convenient and popular WTA mechanism.Today’s CNN-based DL machines profita lot from later CNN work(e.g.,LeCun et al.,1989;Ranzato et al.,2007)(Sec.5.8,5.16,5.19).5.51960-1981and Beyond:Development of Backpropagation(BP)for NNsThe minimisation of errors through gradient descent(Hadamard,1908)in the parameter space of com-plex,nonlinear,differentiable,multi-stage,NN-related systems has been discussed at least since the early 1960s(e.g.,Kelley,1960;Bryson,1961;Bryson and Denham,1961;Pontryagin et al.,1961;Dreyfus,1962; Wilkinson,1965;Amari,1967;Bryson and Ho,1969;Director and Rohrer,1969;Griewank,2012),ini-tially within the framework of Euler-LaGrange equations in the Calculus of Variations(e.g.,Euler,1744). Steepest descent in such systems can be performed(Bryson,1961;Kelley,1960;Bryson and Ho,1969)by iterating the ancient chain rule(Leibniz,1676;L’Hˆo pital,1696)in Dynamic Programming(DP)style(Bell-man,1957).A simplified derivation of the method uses the chain rule only(Dreyfus,1962).The methods of the1960s were already efficient in the DP sense.However,they backpropagated derivative information through standard Jacobian matrix calculations from one“layer”to the previous one, explicitly addressing neither direct links across several layers nor potential additional efficiency gains due to network sparsity(but perhaps such enhancements seemed obvious to the authors).。

随机神经网络发展现状综述一、本文概述随着和机器学习技术的迅猛发展，神经网络已成为一种强大的工具，广泛应用于各种领域，如计算机视觉、语音识别、自然语言处理、游戏等。

其中，随机神经网络作为一种新兴的神经网络架构，近年来引起了广泛的关注和研究。

本文旨在综述随机神经网络的发展现状，包括其基本原理、应用领域、挑战与前景等，以期为读者提供一个全面而深入的了解。

随机神经网络，顾名思义，是一种在神经网络中引入随机性的网络架构。

与传统的深度学习模型相比，随机神经网络在权重初始化、激活函数选择、网络结构等方面具有更高的灵活性和随机性。

这种随机性不仅有助于提升模型的泛化能力，还能在一定程度上解决深度学习模型中的一些固有问题，如过拟合、梯度消失等。

本文首先简要介绍了随机神经网络的基本概念和发展历程，然后重点分析了其在各个应用领域中的表现。

在此基础上，本文还深入探讨了随机神经网络所面临的挑战，如如何平衡随机性与稳定性、如何设计有效的训练算法等。

本文展望了随机神经网络未来的发展趋势和研究方向，以期为推动该领域的发展提供有益的参考。

二、随机神经网络的理论基础随机神经网络（Random Neural Networks, RNNs）的理论基础主要建立在概率论、统计学习理论以及优化算法的基础之上。

其核心思想是通过引入随机性来增强网络的泛化能力和鲁棒性，同时减少过拟合的风险。

在概率论方面，随机神经网络利用随机权重和随机连接来模拟人脑神经元的随机性和不确定性。

这种随机性可以在训练过程中引入噪声，从而提高网络对噪声数据和未知数据的处理能力。

同时，随机性还有助于探索更多的解空间，增加网络的多样性，避免陷入局部最优解。

在统计学习理论方面，随机神经网络通过引入正则化项来控制模型的复杂度，防止过拟合现象的发生。

正则化项通常包括权重衰减、dropout等策略，这些策略可以在训练过程中随机关闭一部分神经元或连接，从而减少网络的复杂度，提高泛化能力。

人工神经网络的最新发展综述摘要:人工神经网络是指模拟人脑神经系统的结构和功能,运用大量的处理部件,由人工方式建立起来的网络系统。

该文首先介绍了神经网络研究动向，然后介绍了近年来几种新型神经网络的基本模型及典型应用,包括模糊神经网络、神经网络与遗传算法的结合、进化神经网络、混沌神经网络和神经网络与小波分析的结合。

最后,根据这几种新型神经网络的特点,展望了它们今后的发展前景。

关键词:模糊神经网络;神经网络与遗传算法的结合;进化神经网络;混沌神经网络；神经网络与小波分析。

The review of the latest developments in artificial neuralnetworksAbstract:Artificial neural network is the system that simulates the human brain’s structure and function, and uses a large number of processing elements, and is manually established by the network system. This paper firstly introduces the research trends of the neural network, and then introduces several new basic models of neural networks and typical applications in recent years, including of fuzzy neural network, the combine of neural network and genetic algorithm, evolutionary neural networks, chaotic neural networks and the combine of neural networks and wavelet analysis. Finally, their future prospects are predicted based on the characteristics of these new neural networks in the paper.Key words: Fuzzy neural network; Neural network and genetic algorithm; Evolutionary neural networks; Chaotic neural networks; Neural networks and wavelet analysis1 引言人工神经网络的研究始于20世纪40年代初。

人工神经网络系统辨识综述摘要：当今社会，系统辨识技术的发展逐渐成熟，人工神经网络的系统辨识方法的应用也越来越多，遍及各个领域。

首先对神经网络系统辨识方法与经典辨识法进行对比，显示出其优越性，然后再通过对改进后的算法具体加以说明，最后展望了神经网络系统辨识法的发展方向。

关键词：神经网络；系统辨识；系统建模0引言随着社会的进步,越来越多的实际系统变成了具有不确定性的复杂系统，经典的系统辨识方法在这些系统中应用，体现出以下的不足:（1）在某些动态系统中,系统的输入常常无法保证，但是最小二乘法的系统辨识法一般要求输入信号已知，且变化较丰富。

（2）在线性系统中，传统的系统辨识方法比在非线性系统辨识效果要好。

（3）不能同时确定系统的结构与参数和往往得不到全局最优解，是传统辨识方法普遍存在的两个缺点。

随着科技的继续发展，基于神经网络的辨识与传统的辨识方法相比较具有以下几个特点:第一，可以省去系统机构建模这一步，不需要建立实际系统的辨识格式；其次，辨识的收敛速度仅依赖于与神经网络本身及其所采用的学习算法，所以可以对本质非线性系统进行辨识;最后可以通过调节神经网络连接权值达到让网络输出逼近系统输出的目的；作为实际系统的辨识模型，神经网络还可用于在线控制。

1神经网络系统辨识法1.1神经网络人工神经网络迅速发展于20世纪末，并广泛地应用于各个领域，尤其是在模式识别、信号处理、工程、专家系统、优化组合、机器人控制等方面。

随着神经网络理论本身以及相关理论和相关技术的不断发展，神经网络的应用定将更加深入。

神经网络，包括前向网络和递归动态网络，将确定某一非线性映射的问题转化为求解优化问题，有一种改进的系统辨识方法就是通过调整网络的权值矩阵来实现这一优化过程。

1.2辨识原理选择一种适合的神经网络模型来逼近实际系统是神经网络用于系统辨识的实质。

其辨识有模型、数据和误差准则三大要素。

系统辨识实际上是一个最优化问题,由辨识的目的与辨识算法的复杂性等因素决定其优化准则。