12a_b (лекции)

Neural Nets12a,12bPeter Savostin,Arthur Martirosyan,Jiang LeiContentnGlossarynBiological modelnModel inspired by our brainnNeural net as function approximatornComplexity of cross connected neural netnDemonstrationnConvolutional Neural NetworksnAutocodingnFinal LayernSoftmaxnDropoutnInference2GlossarynnnnnAxon - аксон, отросток по которому идут нервныеимпульсы к нервным клеткам.Dendrite - дендрит, получает информацию черезсинапсы от аксонов.Synaptic gaps - cинаптические разрывы междуаксонами и дендритами.Refactory period - период рефрактерности, интервал,в течение которого возбудимая ткань не способнагенерировать повторный сигнал.Axonal bifurcation - аксональная бифуркация,разделение аксона на 2 ветви одинакового диаметра,отходящие в стороны под одинаковыми углами.Savostin, Martirosyan3GlossarynnnThreshold box - пороговый блок, нейрон получаетданные от других нейронов, определяет значениекаждого входа и добавляет эти значения.

Если общийвход выше пороговой величины, то выход блока равенединице, в противном случае – нулю.Cumulative influence - совокупное влияниенескольких входов нейронаSynaptic weight - cинаптический вес, в биологиижидкость в синаптических разрывахSavostin, Martirosyan4GlossarynMagnitude of difference - порядок величиныnPerformance function - функция оценки качестваработы нейронной сетиnComputationally intractable - вычислимо сложноnGradient ascent - градиентное восхождениеnChain Rule - правило дифференцирования сложнойфункцииSavostin, Martirosyan5GlossarynDot product - скалярное произведениеnDeep neural nets - глубокие нейронные сети, сеть снесколькими скрытыми слоями нейроновnСonvolutional neural network - сверточная нейронная сетьnPooling - объединение, одна из стадий работы сверточнойнейронной сетиnKernel - ядро, функция применяемая к пикселям некоторогоизображенияnVivinity - окрестности6GlossarynGrid - сеткаnBack propogation - метод обратного распространенияошибкиnGeneralization - обощениеnBoltzmann machines - вид рекуррентных нейронныхсетейnSoftmax - функция, «сжимает» вектор действительныхвеличин до вектора с элементами от 0 до 1nSaddle points - cедловые точки7Biological model1.

All or non2. Cumulative influence3. Synaptic weightSavostin, Martirosyan4.Refractor period5.Axonal bifurcation6.Time patterns8Model inspired by our braininputweights(0,1)""summeroutput∑(0,1)#threshold box#What we can model:1. All or non – threshold box2. Cumulative influence - summer3. Synaptic weight - weightsSavostin, MartirosyanZ – (0,1)TWhat we don’t know how to model:4.Refractor period5.Axonal bifurcation6.Time patterns9Neural net as functionapproximatorOur skull is like a “box full of neurons”, in fact it is betterto say “box full of weights and thresholds”"Tw#Tw"w'Then output is a function of input vector, weight vector and threshold vector̅ = (̅ , 1, 3)When we will train neural net, all we are going be able to do is adjust 1, 3,that’s why neural net is some kind of function approximatorWe can figure out how good our neural net = − ̅ − ̅is by using performance function .It compares the desired value and the actual ̅ = (̅ )value'10Complexity of crossconnected neural netWe can add multiple layers and make cross connectionsbetween nodes.Will it affect the complexity and this network can be growingexponential?"""""∫∫∑∑'';;'88∑';∫Savostin, Martirosyan'"8∑∫'11Complexity of crossconnected neural net=""+'"""''"∑''8∫8∑" " "" " ;' ' "' " ;∑;;'=;"+'" " "" " "' " "" " "∫Savostin, Martirosyan∫"∫'';';;'Width;" Linear in depth8∑12Demonstration13Convolutional NeuralNetworksconvolutionpoolingF(MITE)Savostin, Martirosyan14INPUT [32x32x3] will hold the raw pixel values of the image, in this case an imageof width 32, height 32, and with three color channels R,G,B.CONV layer will compute the output of neurons that are connected to localregions in the input, each computing a dot product between their weights and asmall region they are connected to in the input volume.

This may result in volumesuch as [32x32x12] if we decided to use 12 filters.RELU layer will apply an elementwise activation function, such asthe max(0,x)max(0,x) thresholding at zero. This leaves the size of the volume 15unchanged ([32x32x12]).Autocoding""" = "##' = 'Network has a tight bottleneck of a few neurons in the middle, forcingit to create effective representations that compress the input into a lowdimensional code that can be used by the decoder to reproduce theoriginal input.16Final LayerhiddenlayeroutputlayerIn the final layer, we want to associate aprobability with each output A . Throughsigmoid function, we can get a A (), howcan we get the probability? Just use theformula：A ()A = A =∑CD" C ()So that we can get a probability for eachoutput, it’s a extension of sigmoid function.Jiang LeiSoftmaxSoftmax function is a formula like this:exp(AJ ) = L∑KD" exp(KJ )Through softmax, we can make belongs to range [0,1].

In theclassification problem, is the parameters we want to get to make as big as possible. We have many ways to make belongs to[0,1], why here we use exp()? Refer to logistic function, we can use thisfunction to make the positive result tend to 1 and make the negativeresult tend to 0.Jiang LeiQuestionWhat the relation between softmax and sigmoid function?In fact, softmax is a generalization of the logistic function. As weknow, we can use logistic function to do 2-dimensional classification.In the same way, we can use softmax to do k-dimensionalclassification. So, we can say:Softmax function is a generic sigmoid function.Sigmoid function is a specific softmax function.Jiang LeiDropoutIn the class, this part of explanation is not very clear, we can use anexample to review dropout.Some of us maybe know about MNIST database of handwritten digits.

Wecan train a neural network to recognize them. When we test the model, wedon’t use the whole image(digit), we cover half of the digit, like thefollowing:Jiang LeiHalf-DigitJiang LeiInferenceWe can find that, even if for us humans, it is difficult to identify them. Ifwe cover part of the pictures, how can we identify them? For ushumans, as long as it is not to cover the most critical information,human beings can identify some of the whole object through localinformation. This shows that humans have the ability to use partialinformation for inference.

How can we make the neural network havesuch an ability? It is Dropout.Jiang LeiDropoutAt this time, we may use dropout idea to solve the problem. In thetraining process, I only use part of the neural units, not all units. Thatmeans I will drop out some information in every training step.

Becauseevery time I only use part of the training information, then I naturallyhave the ability of using part of the information to predict, so that ourmodel become more generalization.Jiang LeiQuestionA great question from reddit:Why do I never see dropout applied in convolutional layers?The additional gain in performance obtained by addingdropout in the convolutional layers is worth noting. One mayhave presumed that since the convolutional layers don’t have alot of parameters, overfitting is not a problem and thereforedropout would not have much effect.

However, dropout in thelower layers still helps because it provides noisy inputs for thehigher fully connected layers which prevents them fromoverfitting.Jiang Lei.