# Deep Learning Dictionary - Lightweight Crash Course

Deep Learning Course - Level: Beginner

## Relu Activation Function - Deep Learning Dictionary

### video

expand_more chevron_left

### text

expand_more chevron_left

### ReLU Activation Function - Deep Learning Dictionary

In a neural network, an activation function applies a nonlinear transformation to the output of a layer.

One of the most widely used activation functions today called $$\text{ReLU}$$, short for Rectified Linear Unit, transforms its input to the maximum of either $$0$$ or the input itself.

For a given value $$x$$ passed to $$\text{ReLU}$$, we define

$$\text{relu}(x)=\text{max}(0,x)$$

The table below summarizes how $$\text{ReLU}$$ transforms its input.

Input ReLU Output
Values less than or equal to $$0$$ $$0$$
Values greater than $$0$$ The input value

When $$\text{ReLU}$$ is used as an activation function following a layer in a neural network, it accepts the weighted sum of outputs from the previous layer and transforms this sum to a value equal to either the sum itself or $$0$$.

Intuitively, we can think of a given node's activated output (with $$\text{ReLU}$$) as being "more activated" the more positive it is.

$$\text{ReLU}$$ is by far the most popular choice for an activation function to include in neural networks today. There have been several variations of $$\text{ReLU}$$ that lead to marginal improvements when training networks, like leaky $$\text{ReLU}$$ and parametric $$\text{ReLU}$$ ($$\text{PreLU}$$).

### quiz

expand_more chevron_left

### resources

expand_more chevron_left
What is the relu activation function used in artificial neural networks? π To gain early access to the full Deep Learning Dictionary course, register at: π https://deeplizard.com/course/ddcpailzrd π For more in depth lessons, check out the Deep Learning Fundamentals course: π https://deeplizard.com/course/dlcpailzrd ππ¦ VIDEO SECTIONS π¦π 00:00 Welcome to DEEPLIZARD - Go to deeplizard.com for learning resources 00:10 Relu Activation Function 02:16 Collective Intelligence and the DEEPLIZARD HIVEMIND π₯π¦ DEEPLIZARD COMMUNITY RESOURCES π¦π₯ π Hey, we're Chris and Mandy, the creators of deeplizard! π CHECK OUT OUR VLOG: π https://youtube.com/deeplizardvlog πͺ CHECK OUT OUR FITNESS CHANNEL: π https://www.youtube.com/channel/UCdCxHNCexDrAx78VfAuyKiA π§  Use code DEEPLIZARD at checkout to receive 15% off your first Neurohacker order: π https://neurohacker.com/shop?rfsn=6488344.d171c6 β€οΈπ¦ Special thanks to the following polymaths of the deeplizard hivemind: Mano Prime π Follow deeplizard: Our vlog: https://youtube.com/deeplizardvlog Fitness: https://www.youtube.com/channel/UCdCxHNCexDrAx78VfAuyKiA Facebook: https://facebook.com/deeplizard Instagram: https://instagram.com/deeplizard Twitter: https://twitter.com/deeplizard Patreon: https://patreon.com/deeplizard YouTube: https://youtube.com/deeplizard π Deep Learning with deeplizard: AI Art for Beginners - https://deeplizard.com/course/sdcpailzrd Deep Learning Dictionary - https://deeplizard.com/course/ddcpailzrd Deep Learning Fundamentals - https://deeplizard.com/course/dlcpailzrd Learn TensorFlow - https://deeplizard.com/course/tfcpailzrd Learn PyTorch - https://deeplizard.com/course/ptcpailzrd Natural Language Processing - https://deeplizard.com/course/txtcpailzrd Reinforcement Learning - https://deeplizard.com/course/rlcpailzrd Generative Adversarial Networks - https://deeplizard.com/course/gacpailzrd Stable Diffusion Masterclass - https://deeplizard.com/course/dicpailzrd π Other Courses: DL Fundamentals Classic - https://deeplizard.com/learn/video/gZmobeGL0Yg Deep Learning Deployment - https://deeplizard.com/learn/video/SI1hVGvbbZ4 Data Science - https://deeplizard.com/learn/video/d11chG7Z-xk Trading - https://deeplizard.com/learn/video/ZpfCK_uHL9Y π Check out products deeplizard recommends on Amazon: π https://amazon.com/shop/deeplizard π Get a FREE 30-day Audible trial and 2 FREE audio books using deeplizard's link: π https://amzn.to/2yoqWRn π΅ deeplizard uses music by Kevin MacLeod π https://youtube.com/channel/UCSZXFhRIx6b0dFX3xS8L1yQ β€οΈ Please use the knowledge gained from deeplizard content for good, not evil.

expand_more chevron_left