Deep Learning Dictionary - Lightweight Crash Course

Deep Learning Course 1 of 7 - Level: Beginner

Relu Activation Function - Deep Learning Dictionary

video

expand_more chevron_left

text

expand_more chevron_left

ReLU Activation Function - Deep Learning Dictionary

In a neural network, an activation function applies a nonlinear transformation to the output of a layer.

One of the most widely used activation functions today called \(\text{ReLU}\), short for Rectified Linear Unit, transforms its input to the maximum of either \(0\) or the input itself.

For a given value \(x\) passed to \(\text{ReLU}\), we define

$$\text{relu}(x)=\text{max}(0,x)$$

drawing

The table below summarizes how \(\text{ReLU}\) transforms its input.

Input ReLU Output
Values less than or equal to \(0\) \(0\)
Values greater than \(0\) The input value

When \(\text{ReLU}\) is used as an activation function following a layer in a neural network, it accepts the weighted sum of outputs from the previous layer and transforms this sum to a value equal to either the sum itself or \(0\).

drawing

Intuitively, we can think of a given node's activated output (with \(\text{ReLU}\)) as being "more activated" the more positive it is.

\(\text{ReLU}\) is by far the most popular choice for an activation function to include in neural networks today. There have been several variations of \(\text{ReLU}\) that lead to marginal improvements when training networks, like leaky \(\text{ReLU}\) and parametric \(\text{ReLU}\) (\(\text{PreLU}\)).

quiz

expand_more chevron_left
deeplizard logo DEEPLIZARD Message notifications

Quiz Results

resources

expand_more chevron_left
What is the relu activation function used in artificial neural networks? πŸ‘‰ To gain early access to the full Deep Learning Dictionary course, register at: πŸ”— https://deeplizard.com/course/ddcpailzrd πŸ‘‰ For more in depth lessons, check out the Deep Learning Fundamentals course: πŸ”— https://deeplizard.com/course/dlcpailzrd πŸ•’πŸ¦Ž VIDEO SECTIONS πŸ¦ŽπŸ•’ 00:00 Welcome to DEEPLIZARD - Go to deeplizard.com for learning resources 00:10 Relu Activation Function 02:16 Collective Intelligence and the DEEPLIZARD HIVEMIND πŸ’₯🦎 DEEPLIZARD COMMUNITY RESOURCES 🦎πŸ’₯ πŸ‘‹ Hey, we're Chris and Mandy, the creators of deeplizard! πŸ‘€ CHECK OUT OUR VLOG: πŸ”— https://youtube.com/deeplizardvlog πŸ‘‰ Check out the website for more learning material: πŸ”— https://deeplizard.com πŸ’» ENROLL TO GET DOWNLOAD ACCESS TO CODE FILES πŸ”— https://deeplizard.com/resources 🧠 Support collective intelligence, join the deeplizard hivemind: πŸ”— https://deeplizard.com/hivemind 🧠 Use code DEEPLIZARD at checkout to receive 15% off your first Neurohacker order πŸ‘‰ Use your receipt from Neurohacker to get a discount on deeplizard courses πŸ”— https://neurohacker.com/shop?rfsn=6488344.d171c6 ❀️🦎 Special thanks to the following polymaths of the deeplizard hivemind: Tammy Mano Prime Ling Li πŸš€ Boost collective intelligence by sharing this video on social media! πŸ‘€ Follow deeplizard: Our vlog: https://youtube.com/deeplizardvlog Facebook: https://facebook.com/deeplizard Instagram: https://instagram.com/deeplizard Twitter: https://twitter.com/deeplizard Patreon: https://patreon.com/deeplizard YouTube: https://youtube.com/deeplizard πŸŽ“ Deep Learning with deeplizard: Deep Learning Dictionary - https://deeplizard.com/course/ddcpailzrd Deep Learning Fundamentals - https://deeplizard.com/course/dlcpailzrd Learn TensorFlow - https://deeplizard.com/course/tfcpailzrd Learn PyTorch - https://deeplizard.com/course/ptcpailzrd Natural Language Processing - https://deeplizard.com/course/txtcpailzrd Reinforcement Learning - https://deeplizard.com/course/rlcpailzrd Generative Adversarial Networks - https://deeplizard.com/course/gacpailzrd πŸŽ“ Other Courses: DL Fundamentals Classic - https://deeplizard.com/learn/video/gZmobeGL0Yg Deep Learning Deployment - https://deeplizard.com/learn/video/SI1hVGvbbZ4 Data Science - https://deeplizard.com/learn/video/d11chG7Z-xk Trading - https://deeplizard.com/learn/video/ZpfCK_uHL9Y πŸ›’ Check out products deeplizard recommends on Amazon: πŸ”— https://amazon.com/shop/deeplizard 🎡 deeplizard uses music by Kevin MacLeod πŸ”— https://youtube.com/channel/UCSZXFhRIx6b0dFX3xS8L1yQ ❀️ Please use the knowledge gained from deeplizard content for good, not evil.

updates

expand_more chevron_left
deeplizard logo DEEPLIZARD Message notifications

Update history for this page

Did you know you that deeplizard content is regularly updated and maintained?

  • Updated
  • Maintained

Spot something that needs to be updated? Don't hesitate to let us know. We'll fix it!


All relevant updates for the content on this page are listed below.