fishy.models.deep.moe

Mixture of Experts (MoE) model for spectral data.

This model uses a gating network to dynamically weight the outputs of multiple expert networks (Transformers) based on the input spectrum.

Classes

class fishy.models.deep.moe.MixtureOfExperts(input_dim: int, output_dim: int, hidden_dim: int = 128, num_layers: int = 4, num_heads: int = 4, dropout: float = 0.1, num_experts: int = 3, **kwargs)

Bases: Module

Mixture of Experts (MoE) using Transformer experts.

experts

List of expert Transformer models.

Type:

nn.ModuleList

gate

Gating network to determine weights for each expert.

Type:

nn.Linear

__init__(input_dim: int, output_dim: int, hidden_dim: int = 128, num_layers: int = 4, num_heads: int = 4, dropout: float = 0.1, num_experts: int = 3, **kwargs) → None

Initializes the MixtureOfExperts model.

Parameters:

• input_dim (int) – Number of input features.

• output_dim (int) – Number of output classes/dimensions.

• hidden_dim (int, optional) – Hidden dimension. Defaults to 128.

• num_layers (int, optional) – Layers per expert. Defaults to 4.

• num_heads (int, optional) – Heads per expert. Defaults to 4.

• dropout (float, optional) – Dropout rate. Defaults to 0.1.

• num_experts (int, optional) – Number of expert networks. Defaults to 3.

forward(x: Tensor, *args, **kwargs) → Tensor

Forward pass.

s