These are generic choices used as aplace to begin learning, I wasn't picking them to be special in anyway. To make things definite, I'll pick the initial weight to be $0.6$ andthe initial bias to be $0.9$. So let's take a look at how the neuron learns. However, it turns out to beilluminating to use gradient descent to attempt to learn a weight andbias. Of course, this is such a trivial taskthat we could easily figure out an appropriate weight and bias byhand, without using a learning algorithm. We'll train this neuron to do something ridiculously easy: take theinput $1$ to the output $0$. The example involves a neuronwith just one input: Is this what happens in practice? To answer thisquestion, let's look at a toy example. Ideally, we hope and expect that our neural networks will learn fastfrom their errors. You can bet that the next time I playedbefore an audience I played in the correct octave! By contrast, welearn more slowly when our errors are less well-defined. Yet while unpleasant, we also learn quickly whenwe're decisively wrong. I gotconfused, and couldn't continue until someone pointed out my error. I wasnervous, and began playing the piece an octave too low. Soon after beginning tolearn the piano I gave my first performance before an audience.

Most of us find it unpleasant to be wrong. That will leave you well prepared to quickly pick upother techniques, as you need them. Mastering thoseimportant techniques is not just useful in its own right, but willalso deepen your understanding of what problems can arise when you useneural networks. The philosophy isthat the best entree to the plethora of available techniques isin-depth study of a few of the most important. Of course, we're only covering a few of the many, many techniqueswhich have been developed for use in neural nets. We'll also implementmany of the techniques in running code, and use them to improve theresults obtained on the handwriting classification problem studied in Chapter 1. The discussions are largely independentof one another, and so you may jump ahead if you wish. The techniques we'll develop in this chapter include: a better choiceof cost function, known as the cross-entropy cost function four so-called 'regularization' methods (L1 and L2 regularization, dropout, and artificialexpansion of the training data), which make our networks better atgeneralizing beyond the training data a better method for initializing the weights in the network and a set of heuristics to help choose good hyper-parameters for the network.I'll also overview several other techniques in less depth. In this chapter I explain a suite of techniqueswhich can be used to improve on our vanilla implementation ofbackpropagation, and so improve the way our networks learn. In a similar way, up tonow we've focused on understanding the backpropagation algorithm.It's our 'basic swing', the foundation for learning in most work onneural networks. Only gradually do theydevelop other shots, learning to chip, draw and fade the ball,building on and modifying their basic swing. When a golf player is first learning to play golf, they usually spendmost of their time developing a basic swing. Goodfellow, Yoshua Bengio, and Aaron Courville Michael Nielsen's project announcement mailing list Thanks to all the supporters who made the book possible, withД®special thanks to Pavel Dudrenov.

Deep Learning Workstations, Servers, and Laptops

Cross entropy loss function