Pointwise nonlinearities used inside blocks and heads.
Registry: praxis.ACTIVATION_REGISTRY (35 entries)
Original Implementation of the GELU activation function in Google BERT repo when initially created. For information: OpenAI GPT's GELU is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in nn.functional Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415
Source: transformers.activations (external dependency)
Value: (<class 'transformers.activations.ClippedGELUActivation'>, {'min': -10, 'max': 10})
Applies GELU approximation that is faster than default and more accurate than QuickGELU. See: https://github.com/hendrycks/GELUs
Implemented along with MEGA (Moving Average Equipped Gated Attention)
Source: transformers.activations (external dependency)
Applies GELU approximation that is slower than QuickGELU but more accurate. See: https://github.com/hendrycks/GELUs
Source: transformers.activations (external dependency)
Implementation of the GELU activation function currently in Google BERT repo (identical to OpenAI GPT). Also see the Gaussian Error Linear Units paper: https://huggingface.co/papers/1606.08415
Source: transformers.activations (external dependency)
Value: (<class 'transformers.activations.GELUActivation'>, {'use_gelu_python': True})
Value: (<class 'transformers.activations.GELUTanh'>, {'use_gelu_tanh_python': True})
A fast C implementation of the tanh approximation of the GeLU activation function. See https://huggingface.co/papers/1606.08415.
This implementation is equivalent to NewGELU and FastGELU but much faster. However, it is not an exact numerical match due to rounding errors.
Source: transformers.activations (external dependency)
Applies the Hardswish function, element-wise.
Method described in the paper: Searching for MobileNetV3 <https://arxiv.org/abs/1905.02244>_.
Hardswish is defined as:
.. math:: \text{Hardswish}(x) = \begin{cases} 0 & \text{if~} x \le -3, \ x & \text{if~} x \ge +3, \ x \cdot (x + 3) /6 & \text{otherwise} \end{cases}
Source: torch.nn.modules.activation (external dependency)
Sum of sines at fixed frequencies and amplitudes.
Returns sum(a_i * sin(f_i * x)) for buffer-registered (f_i, a_i) pairs. A cheap
fixed-spectrum periodic activation - no learnable params.
Source: praxis/activations/jagged_sine.py:7
Applies elementwise activation based on Laplace function, introduced in MEGA as an attention activation. See https://huggingface.co/papers/2209.10655
Inspired by squared relu, but with bounded range and gradient for better stability
Source: transformers.activations (external dependency)
Applies the LeakyReLU function element-wise.
.. math:: \text{LeakyReLU}(x) = \max(0, x) + \text{negative_slope} * \min(0, x)
or
.. math:: \text{LeakyReLU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \ \text{negative_slope} \times x, & \text{ otherwise } \end{cases}
Source: torch.nn.modules.activation (external dependency)
Applies the linear activation function, i.e. forwarding input directly to output.
Source: transformers.activations (external dependency)
See Mish: A Self-Regularized Non-Monotonic Activation Function (Misra., https://huggingface.co/papers/1908.08681). Also visit the official repository for the paper: https://github.com/digantamisra98/Mish
Source: transformers.activations (external dependency)
Implements NMDA - an activation function which mimics N-methyl-D-aspartic acid receptors (NMDAR) in the brain. NMDAR-like nonlinearity shifts short-term working memory into long-term reference memory, thus enhancing a process that is similar to memory consolidation in the mammalian brain. https://openreview.net/forum?id=vKpVJxplmB
Source: praxis/activations/nmda.py:8
Serpent applied recurrently, with a per-feature, per-token gate that lets a feature stop iterating once it has converged.
h_0 = 0, open_0 = 1 a_eff = a * (1 + MOD_MAX * tanh(w) * h_k) y = serpent(x, a_eff, b, g) logit = u_k + p * m(x) + q * conv(x, y) z_k = hard_concrete(logit) open_k = open_{k-1} * z_k # closed stays closed x = x + open_k * (y - x) h_{k+1}= tanh(h_k + (y - x))
WHY THE GATE IS THE POINT. A pointwise map applied N ...
Source: praxis/activations/ouroboros.py:103
Stolen from here: https://github.com/AaltoML/PeriodicBNN/blob/main/python_codes/model.py
Source: praxis/activations/periodic_relu.py:9
Applies the element-wise PReLU function.
.. math:: \text{PReLU}(x) = \max(0,x) + a * \min(0,x)
or
.. math:: \text{PReLU}(x) = \begin{cases} x, & \text{ if } x \ge 0 \ ax, & \text{ otherwise } \end{cases}
Here :math:a is a learnable parameter. When called without arguments, nn.PReLU()
uses a single parameter :math:a across all input channels. If called with
nn.PReLU(nChannels), a separate :math:a is used for each input channel.
.. note:: weight decay ...
Source: torch.nn.modules.activation (external dependency)
Applies GELU approximation that is fast but somewhat inaccurate. See: https://github.com/hendrycks/GELUs
Source: transformers.activations (external dependency)
Applies the rectified linear unit function element-wise.
:math:\text{ReLU}(x) = (x)^+ = \max(0, x)
Source: torch.nn.modules.activation (external dependency)
Applies the relu^2 activation introduced in https://huggingface.co/papers/2109.08668
Source: transformers.activations (external dependency)
Applies the ReLU6 function element-wise.
.. math:: \text{ReLU6}(x) = \min(\max(0,x), 6)
Source: torch.nn.modules.activation (external dependency)
Implements the SERF activation function, as described in: https://arxiv.org/abs/2108.09598
Source: praxis/activations/serf.py:6
Praxis' extended Snake activation with a second oscillation term:
y = x + sin^2(α·x) · α / (α^2 + ε^2) + γ·sin(βx)
α controls the primary squared-sine frequency (original Snake term). β and γ add a secondary sine with its own frequency and amplitude. All three are per-feature learnable parameters.
The 1/α factor in the original Snake is replaced by the smooth-rectified α / (α^2 + ε^2): matches 1/α for |α| >> ε, bounded by 1/ε for |α| ~ 0. Prevents the tiny-α
feature explosion ...
Source: praxis/activations/serpent.py:17
Serpent with a test-time-modulated frequency: a learnable chirp.
s = rms(x, over features) # live per-token energy m = tanh(log(s) - log_s_ref) # centered test-time signal in (-1, 1) a_eff = a
- (1 + MOD_MAX * tanh(v) * m) # frequency breathes with energy y = x + sin^2(a_effx) * a_eff/(a_eff^2 + eps^2) + gsin(b*x)
Serpent learns a static per-feature frequency a. Servant lets that frequency move
at inference with the energy of ...
Source: praxis/activations/servant.py:16
Applies the Sigmoid function element-wise.
.. math:: \text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)}
Shape:
- Input: :math:(*), where :math:* means any number of dimensions.
- Output: :math:(*), same shape as the input.
.. image:: ../scripts/activation_images/Sigmoid.png
Examples::
m = nn.Sigmoid() >>> input = torch.randn(2) >>> output = m(input)
Source: torch.nn.modules.activation (external dependency)
See Gaussian Error Linear Units (Hendrycks et al., https://arxiv.org/abs/1606.08415) where the SiLU (Sigmoid Linear Unit) was originally introduced and coined, and see Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning (Elfwing et al., https://arxiv.org/abs/1702.03118) and Swish: a Self-Gated Activation Function (Ramachandran et al., https://arxiv.org/abs/1710.05941v1) where the SiLU was experimented with later.
Source: transformers.activations (external dependency)
Plain sqrt(2) * sin(x) activation. The constant preserves unit output variance for
unit-variance input, matching SIREN's init scheme.
Source: praxis/activations/sin.py:7
scale * (sin(x) + cos(x)) - a phase-shifted sinusoid that gives the network a non-
zero gradient at the origin, unlike pure sin.
Source: praxis/activations/sin_cos.py:5
Implements SinLU, which has an interesting shape and learnable parameters: https://www.mdpi.com/2227-7390/10/3/337
Source: praxis/activations/sinlu.py:6
Snake activation x + sin^2(a*x) / a with a learnable per-feature frequency a
(https://arxiv.org/abs/2006.08195). a is lazily materialized on first forward to
match the input's feature dimension.
Source: praxis/activations/snake.py:41
sqrt(softplus(x)) — the router scoring function used by DeepSeek V4.
Source: transformers.activations (external dependency)
Applies the Sigmoid Linear Unit (SiLU) function, element-wise.
The SiLU function is also known as the swish function.
.. math:: \text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.}
.. note:: See Gaussian Error Linear Units (GELUs) <https://arxiv.org/abs/1606.08415>_ where the SiLU (Sigmoid Linear Unit) was
originally coined, and see `Sigmoid-Weighted Linear Units for Neural Network
Function Approximation in Reinforcement Learning ...
Source: torch.nn.modules.activation (external dependency)
Applies the Hyperbolic Tangent (Tanh) function element-wise.
Tanh is defined as:
.. math:: \text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)} {\exp(x) + \exp(-x)}
Shape:
- Input: :math:(*), where :math:* means any number of dimensions.
- Output: :math:(*), same shape as the input.
.. image:: ../scripts/activation_images/Tanh.png
Examples::
m = nn.Tanh() >>> input = torch.randn(2) >>> output = m(input)
Source: torch.nn.modules.activation (external dependency)
Applies the xIELU activation function introduced in https://arxiv.org/abs/2411.13010
If the user has installed the nickjbrowning/XIELU wheel, we import xIELU CUDA Otherwise, we emit a single warning and use xIELU Python
Source: transformers.activations (external dependency)