# Deep Learning Dictionary - Lightweight Crash Course

Deep Learning Course - Level: Beginner

## Activation Functions - Deep Learning Dictionary

### video

expand_more chevron_left

### text

expand_more chevron_left

### Activation Functions - Deep Learning Dictionary

In a neural network, an activation function applies a nonlinear transformation to the output of a layer.

Activation functions are biologically inspired by activity in our brains where different neurons fire (or are activated) by different stimuli.

For any given node in a fully connected layer, recall that the node's value is calculated as the weighted sum of its inputs received from the previous layer. After taking this weighted sum, we pass the result to an activation function. This final result is passed as input to the following layer.

The activation function does an operation on this sum to transform it to a value that is often times bounded between some lower and/or upper limit. A key point of this transformation is that it's non-linear.

An important feature of linear functions is that the composition of two linear functions is also a linear function. This means that, even in very deep neural networks, if we only had linear transformations of our data values during a forward pass, the learned mapping in our network from input to output would also be linear.

Typically, the types of mappings that we are aiming to learn with our deep neural networks are more complex than simple linear mappings. Having non-linear activation functions allows our neural networks to compute arbitrarily complex functions.

There are many variations of activation functions, where each transform data differently and are used under different scenarios. Some of the more popular ones include:

• ReLU
• Sigmoid
• Softmax

We'll elaborate on how each of these functions transform data in later lessons.

### quiz

expand_more chevron_left
DEEPLIZARD Message notifications

### resources

expand_more chevron_left
What are activation functions within artificial neural networks? π To gain early access to the full Deep Learning Dictionary course, register at: π https://deeplizard.com/course/ddcpailzrd π For more in depth lessons, check out the Deep Learning Fundamentals course: π https://deeplizard.com/course/dlcpailzrd ππ¦ VIDEO SECTIONS π¦π 00:00 Welcome to DEEPLIZARD - Go to deeplizard.com for learning resources 00:10 Activation Functions 04:50 Collective Intelligence and the DEEPLIZARD HIVEMIND π₯π¦ DEEPLIZARD COMMUNITY RESOURCES π¦π₯ π Hey, we're Chris and Mandy, the creators of deeplizard! π CHECK OUT OUR VLOG: π https://youtube.com/deeplizardvlog πͺ CHECK OUT OUR FITNESS CHANNEL: π https://www.youtube.com/channel/UCdCxHNCexDrAx78VfAuyKiA π§  Use code DEEPLIZARD at checkout to receive 15% off your first Neurohacker order: π https://neurohacker.com/shop?rfsn=6488344.d171c6 β€οΈπ¦ Special thanks to the following polymaths of the deeplizard hivemind: Mano Prime π Follow deeplizard: Our vlog: https://youtube.com/deeplizardvlog Fitness: https://www.youtube.com/channel/UCdCxHNCexDrAx78VfAuyKiA Facebook: https://facebook.com/deeplizard Instagram: https://instagram.com/deeplizard Twitter: https://twitter.com/deeplizard Patreon: https://patreon.com/deeplizard YouTube: https://youtube.com/deeplizard π Deep Learning with deeplizard: AI Art for Beginners - https://deeplizard.com/course/sdcpailzrd Deep Learning Dictionary - https://deeplizard.com/course/ddcpailzrd Deep Learning Fundamentals - https://deeplizard.com/course/dlcpailzrd Learn TensorFlow - https://deeplizard.com/course/tfcpailzrd Learn PyTorch - https://deeplizard.com/course/ptcpailzrd Natural Language Processing - https://deeplizard.com/course/txtcpailzrd Reinforcement Learning - https://deeplizard.com/course/rlcpailzrd Generative Adversarial Networks - https://deeplizard.com/course/gacpailzrd Stable Diffusion Masterclass - https://deeplizard.com/course/dicpailzrd π Other Courses: DL Fundamentals Classic - https://deeplizard.com/learn/video/gZmobeGL0Yg Deep Learning Deployment - https://deeplizard.com/learn/video/SI1hVGvbbZ4 Data Science - https://deeplizard.com/learn/video/d11chG7Z-xk Trading - https://deeplizard.com/learn/video/ZpfCK_uHL9Y π Check out products deeplizard recommends on Amazon: π https://amazon.com/shop/deeplizard π Get a FREE 30-day Audible trial and 2 FREE audio books using deeplizard's link: π https://amzn.to/2yoqWRn π΅ deeplizard uses music by Kevin MacLeod π https://youtube.com/channel/UCSZXFhRIx6b0dFX3xS8L1yQ β€οΈ Please use the knowledge gained from deeplizard content for good, not evil.

### updates

expand_more chevron_left
DEEPLIZARD Message notifications

#### Update history for this page

Did you know you that deeplizard content is regularly updated and maintained?

• Updated
• Maintained

Spot something that needs to be updated? Don't hesitate to let us know. We'll fix it!

All relevant updates for the content on this page are listed below.

Committed by on