Learning Options

Typed options for the RL runners. A task's runner_options holds one; Eden translates it into the configuration dict the installed library version expects, so a task on rsl_rl's own algorithms trains on every rsl-rl-lib version the pin admits (>=3.0). Some of Eden's own algorithms need newer: AMP-PPO and RMA need 5.5.1 (MIN_RSL_RL_VERSION in eden.learning.wrappers.rsl_rl), and the flashsac extra 5.4.2, while HIM adapts itself to 3.x (see Across rsl-rl-lib versions). The training runs guide explains runners, checkpoints and resuming; the algorithms Eden adds on top of a library are on the learning page.

from eden.options.learning.rsl_rl.rl import RslRlOnPolicyRunnerOptions, RslRlPpoAlgorithmOptions

rsl_rl

Models, distributions and PPO

rsl_rl model, distribution, and PPO configuration options.

Attributes

RslRlActorOptions module-attribute

RslRlActorOptions = RslRlModelWithDistributionOptions

RslRlCriticOptions module-attribute

RslRlCriticOptions = RslRlModelOptions

RslRlCriticRNNOptions module-attribute

RslRlCriticRNNOptions = RslRlRNNModelOptions

RslRlCriticCNNOptions module-attribute

RslRlCriticCNNOptions = RslRlCNNModelOptions

Classes

RslRlDistributionOptions

RslRlDistributionOptions(**data)

Bases: ConfigurableOptions

Base configuration for an output distribution.

PARAMETER	DESCRIPTION
class_name	The distribution class name.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'GaussianDistribution'

RslRlGaussianDistributionOptions

RslRlGaussianDistributionOptions(**data)

Bases: RslRlDistributionOptions

Configuration for a state-independent Gaussian distribution.

PARAMETER	DESCRIPTION
class_name	The distribution class name.
init_std	Initial standard deviation.
std_type	Parameterisation of the standard deviation.
std_range	Hard (min, max) clamp on the standard deviation. The floor is what keeps a policy exploring once its advantage signal flattens out -- without one, a task whose reward briefly stops depending on the action can drive the std to zero and never recover. Defaults to (1e-6, 1e6), rsl-rl's own effectively-unclamped range -- a concrete default rather than None because rsl-rl splats this whole dict into the distribution constructor (dist_class(output_dim, **dist_cfg)), where a None placeholder raises 'NoneType' object is not iterable instead of meaning "unset". rsl-rl >= 5 only. The v3.2 and v4 translators (:mod:eden.options.learning.rsl_rl.translate) forward only init_std and std_type, so on an older rsl-rl this field is dropped and the clamp simply does not apply -- set it as insurance, not as something a run depends on.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'GaussianDistribution'

init_std class-attribute instance-attribute

init_std: float = 1.0

std_type class-attribute instance-attribute

std_type: Literal['scalar', 'log'] = 'log'

std_range class-attribute instance-attribute

std_range: tuple[float, float] = (1e-06, 1000000.0)

RslRlHeteroscedasticGaussianDistributionOptions

RslRlHeteroscedasticGaussianDistributionOptions(**data)

Bases: RslRlDistributionOptions

Configuration for a state-dependent (heteroscedastic) Gaussian distribution.

The MLP outputs both mean and standard deviation. The init_std value is used to initialise the bias of the std head.

PARAMETER	DESCRIPTION
class_name	The distribution class name.
init_std	Initial standard deviation (used to initialise MLP std-head bias).
std_type	Parameterisation of the standard deviation.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'HeteroscedasticGaussianDistribution'

init_std class-attribute instance-attribute

init_std: float = 1.0

std_type class-attribute instance-attribute

std_type: Literal['scalar', 'log'] = 'scalar'

RslRlModelOptions

RslRlModelOptions(**data)

Bases: ConfigurableOptions

Configuration for an rsl_rl model.

PARAMETER	DESCRIPTION
class_name	The rsl_rl class name. This parameter should not be overridden.
hidden_dims	Hidden layer dimensions for the MLP trunk.
activation	Activation function.
obs_normalization	Whether to normalize observations before the network.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'MLPModel'

hidden_dims class-attribute instance-attribute

hidden_dims: list[int] = [512, 256, 128]

activation class-attribute instance-attribute

activation: str = 'elu'

obs_normalization class-attribute instance-attribute

obs_normalization: bool = True

RslRlModelWithDistributionOptions

RslRlModelWithDistributionOptions(**data)

Bases: RslRlModelOptions

Mixin for models with a distribution.

PARAMETER	DESCRIPTION
distribution_cfg	Output distribution. None means a deterministic head.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

distribution_cfg class-attribute instance-attribute

distribution_cfg: RslRlDistributionOptions | None = RslRlGaussianDistributionOptions()

RslRlRNNModelOptions

RslRlRNNModelOptions(**data)

Bases: RslRlModelOptions

Configuration for a recurrent (RNN) model.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'RNNModel'

rnn_type class-attribute instance-attribute

rnn_type: str = 'gru'

rnn_hidden_dim class-attribute instance-attribute

rnn_hidden_dim: int = 256

rnn_num_layers class-attribute instance-attribute

rnn_num_layers: int = 1

RslRlCNNModelOptions

RslRlCNNModelOptions(**data)

Bases: RslRlModelOptions

Configuration for a CNN-backed model.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'CNNModel'

cnn_cfg class-attribute instance-attribute

cnn_cfg: dict[str, dict] | dict[str, Any] | None = None

RslRlActorRNNOptions

RslRlActorRNNOptions(**data)

Bases: RslRlRNNModelOptions, RslRlModelWithDistributionOptions

Recurrent policy actor.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

RslRlActorCNNOptions

RslRlActorCNNOptions(**data)

Bases: RslRlCNNModelOptions, RslRlModelWithDistributionOptions

CNN policy actor.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

RslRlPpoAlgorithmOptions

RslRlPpoAlgorithmOptions(**data)

Bases: ConfigurableOptions

Configuration for the PPO algorithm.

PARAMETER	DESCRIPTION
class_name	The algorithm class name.
num_learning_epochs	The number of learning epochs per update.
num_mini_batches	The number of mini-batches per update.
learning_rate	The learning rate for the policy.
schedule	The learning rate schedule.
gamma	The discount factor.
lam	The lambda parameter for Generalized Advantage Estimation (GAE).
entropy_coef	The coefficient for the entropy loss.
desired_kl	The desired KL divergence.
max_grad_norm	The maximum gradient norm.
value_loss_coef	The coefficient for the value loss.
use_clipped_value_loss	Whether to use clipped value loss.
clip_param	The clipping parameter for the policy.
normalize_advantage_per_mini_batch	Whether to normalize the advantage per mini-batch.
share_cnn_encoders	Whether to share CNN encoders between actor and critic.
rnd_cfg	The RND configuration.
symmetry_cfg	The symmetry configuration.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'PPO'

num_learning_epochs class-attribute instance-attribute

num_learning_epochs: int = 5

num_mini_batches class-attribute instance-attribute

num_mini_batches: int = 4

learning_rate class-attribute instance-attribute

learning_rate: float = 0.001

schedule class-attribute instance-attribute

schedule: str = 'adaptive'

gamma class-attribute instance-attribute

gamma: float = 0.99

lam class-attribute instance-attribute

lam: float = 0.95

entropy_coef class-attribute instance-attribute

entropy_coef: float = 0.005

desired_kl class-attribute instance-attribute

desired_kl: float = 0.01

max_grad_norm class-attribute instance-attribute

max_grad_norm: float = 1.0

value_loss_coef class-attribute instance-attribute

value_loss_coef: float = 1.0

use_clipped_value_loss class-attribute instance-attribute

use_clipped_value_loss: bool = True

clip_param class-attribute instance-attribute

clip_param: float = 0.2

normalize_advantage_per_mini_batch class-attribute instance-attribute

normalize_advantage_per_mini_batch: bool = False

share_cnn_encoders class-attribute instance-attribute

share_cnn_encoders: bool = False

rnd_cfg class-attribute instance-attribute

rnd_cfg: RslRlRndOptions | None = None

symmetry_cfg class-attribute instance-attribute

symmetry_cfg: RslRlSymmetryOptions | None = None

RslRlBaseRunnerOptions

RslRlBaseRunnerOptions(**data)

Bases: ConfigurableOptions

Base configuration of the runner.

PARAMETER	DESCRIPTION
seed	The seed for the experiment. Non-negative; see eden.utils.random.validate_seed.
device	The device for the rl-agent.
num_steps_per_env	The number of steps per environment per update.
max_iterations	The maximum number of iterations.
obs_groups	A mapping from observation sets to observation groups. Keys are "actor", "critic" (and optionally "rnd_state"). Values are lists of observation group names provided by the environment.
clip_actions	Actions are clamped to [-clip_actions, clip_actions]; a finite positive bound, or None to leave them unclamped.
save_interval	The number of iterations between saves.
keep_last_checkpoints	How many of the run's newest model_<iteration>.pt checkpoints to keep on disk; each save deletes the older ones. Defaults to 1, the newest, which is all eden train --resume needs; None keeps every checkpoint. Checkpoints already uploaded to the logger (wandb) stay there.
check_for_nan	Whether to check for NaN values in environment outputs.
experiment_name	The experiment name.
run_name	The run name.
logger	The logger to use.
neptune_project	The neptune project name.
wandb_project	The wandb project name.
resume	Whether to resume a previous training.
load_run	The run directory to load.
load_checkpoint	The checkpoint file to load.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

seed class-attribute instance-attribute

seed: Seed = 7100

device class-attribute instance-attribute

device: DeviceStr = 'auto'

num_steps_per_env class-attribute instance-attribute

num_steps_per_env: int = 24

max_iterations class-attribute instance-attribute

max_iterations: int = 30000

obs_groups class-attribute instance-attribute

obs_groups: dict[str, list[str]] = {}

clip_actions class-attribute instance-attribute

clip_actions: ClipActions | None = None

save_interval class-attribute instance-attribute

save_interval: int = 500

keep_last_checkpoints class-attribute instance-attribute

keep_last_checkpoints: int | None = 1

check_for_nan class-attribute instance-attribute

check_for_nan: bool = True

experiment_name class-attribute instance-attribute

experiment_name: str = ''

run_name class-attribute instance-attribute

run_name: str = ''

logger class-attribute instance-attribute

logger: Literal['tensorboard', 'neptune', 'wandb'] = 'wandb'

neptune_project class-attribute instance-attribute

neptune_project: str = 'eden'

wandb_project class-attribute instance-attribute

wandb_project: str = 'eden'

resume class-attribute instance-attribute

resume: bool = False

load_run class-attribute instance-attribute

load_run: str = '.*'

load_checkpoint class-attribute instance-attribute

load_checkpoint: str = 'model_.*.pt'

RslRlOnPolicyRunnerOptions

RslRlOnPolicyRunnerOptions(**data)

Bases: RslRlBaseRunnerOptions

Configuration of the runner for on-policy algorithms.

The runner carries separate actor and critic model configs.

PARAMETER	DESCRIPTION
class_name	The runner class name.
actor	The actor model configuration with distribution to sample from.
critic	The critic model configuration.
algorithm	The algorithm configuration.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'OnPolicyRunner'

actor instance-attribute

actor: RslRlModelWithDistributionOptions

critic instance-attribute

critic: RslRlModelOptions

algorithm instance-attribute

algorithm: RslRlPpoAlgorithmOptions

Distillation

rsl_rl distillation algorithm and runner configuration options.

Classes

RslRlDistillationAlgorithmOptions

RslRlDistillationAlgorithmOptions(**data)

Bases: ConfigurableOptions

Configuration for the distillation algorithm.

PARAMETER	DESCRIPTION
class_name	The algorithm class name.
num_learning_epochs	The number of updates performed with each sample.
learning_rate	The learning rate for the student policy.
gradient_length	The number of environment steps the gradient flows back.
max_grad_norm	The maximum norm the gradient is clipped to.
optimizer	The optimizer to use.
loss_type	The loss type.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'Distillation'

num_learning_epochs class-attribute instance-attribute

num_learning_epochs: int = 1

learning_rate class-attribute instance-attribute

learning_rate: float = 0.001

gradient_length class-attribute instance-attribute

gradient_length: int = 1

max_grad_norm class-attribute instance-attribute

max_grad_norm: float | None = None

optimizer class-attribute instance-attribute

optimizer: Literal['adam', 'adamw', 'sgd', 'rmsprop'] = 'adam'

loss_type class-attribute instance-attribute

loss_type: Literal['mse', 'huber'] = 'mse'

RslRlDistillationRunnerOptions

RslRlDistillationRunnerOptions(**data)

Bases: RslRlBaseRunnerOptions

Configuration of the runner for distillation algorithms.

PARAMETER	DESCRIPTION
class_name	The runner class name.
student	The student model configuration with distribution to sample from.
teacher	The teacher model configuration.
algorithm	The algorithm configuration.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

class_name class-attribute instance-attribute

class_name: str = 'DistillationRunner'

student instance-attribute

student: RslRlModelWithDistributionOptions

teacher instance-attribute

teacher: RslRlModelOptions

algorithm instance-attribute

algorithm: RslRlDistillationAlgorithmOptions

Random network distillation

rsl_rl Random Network Distillation (RND) configuration options.

Classes

RslRlRndWeightScheduleOptions

RslRlRndWeightScheduleOptions(**data)

Bases: ConfigurableOptions

Configuration for the weight schedule.

PARAMETER	DESCRIPTION
mode	The type of weight schedule.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

mode class-attribute instance-attribute

mode: str = 'constant'

RslRlRndLinearWeightScheduleOptions

RslRlRndLinearWeightScheduleOptions(**data)

Bases: RslRlRndWeightScheduleOptions

Configuration for the linear weight schedule.

This schedule decays the weight linearly from the initial value to the final value between initial_step and before final_step.

PARAMETER	DESCRIPTION
mode	The type of weight schedule.
final_value	The final value of the weight parameter.
initial_step	The initial step of the weight schedule. For steps before this step, the weight is the initial value specified in RslRlRndOptions.weight.
final_step	The final step of the weight schedule. For steps after this step, the weight is the final value specified in final_value.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

mode class-attribute instance-attribute

mode: str = 'linear'

final_value instance-attribute

final_value: float

initial_step instance-attribute

initial_step: int

final_step instance-attribute

final_step: int

RslRlRndStepWeightScheduleOptions

RslRlRndStepWeightScheduleOptions(**data)

Bases: RslRlRndWeightScheduleOptions

Configuration for the step weight schedule.

This schedule sets the weight to the value specified in final_value at step final_step.

PARAMETER	DESCRIPTION
mode	The type of weight schedule.
final_step	The final step of the weight schedule. For steps after this step, the weight is the value specified in final_value.
final_value	The final value of the weight parameter.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

mode class-attribute instance-attribute

mode: str = 'step'

final_step instance-attribute

final_step: int

final_value instance-attribute

final_value: float

RslRlRndOptions

RslRlRndOptions(**data)

Bases: ConfigurableOptions

Configuration for the Random Network Distillation (RND) module.

For more information, please check the work from :cite:schwarke2023curiosity.

PARAMETER	DESCRIPTION
weight	The weight for the RND reward (also known as intrinsic reward). Similar to other reward terms, the RND reward is scaled by this weight.
weight_schedule	The weight schedule for the RND reward. Default is None, which means the weight is constant.
reward_normalization	Whether to normalize the RND reward.
state_normalization	Whether to normalize the RND state.
learning_rate	The learning rate for the RND module.
num_outputs	The number of outputs for the RND module.
predictor_hidden_dims	The hidden dimensions for the RND predictor network. If the list contains -1, then the hidden dimensions are the same as the input dimensions.
target_hidden_dims	The hidden dimensions for the RND target network. If the list contains -1, then the hidden dimensions are the same as the input dimensions.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

weight class-attribute instance-attribute

weight: float = 0.0

weight_schedule class-attribute instance-attribute

weight_schedule: RslRlRndWeightScheduleOptions | None = None

reward_normalization class-attribute instance-attribute

reward_normalization: bool = False

state_normalization class-attribute instance-attribute

state_normalization: bool = False

learning_rate class-attribute instance-attribute

learning_rate: float = 0.001

num_outputs class-attribute instance-attribute

num_outputs: int = 1

predictor_hidden_dims class-attribute instance-attribute

predictor_hidden_dims: list[int] = [-1]

target_hidden_dims class-attribute instance-attribute

target_hidden_dims: list[int] = [-1]

Symmetry augmentation

rsl_rl symmetry-augmentation configuration options.

Classes

RslRlSymmetryOptions

RslRlSymmetryOptions(**data)

Bases: ConfigurableOptions

Configuration for the symmetry-augmentation in the training.

When use_data_augmentation is True, the data_augmentation_func is used to generate augmented observations and actions. These are then used to train the model.

When use_mirror_loss is True, the mirror_loss_coeff is used to weight the symmetry-mirror loss. This loss is directly added to the agent's loss function.

If both use_data_augmentation and use_mirror_loss are False, then no symmetry-based training is enabled. However, the data_augmentation_func is called to compute and log symmetry metrics. This is useful for performing ablations.

For more information, please check the work from :cite:mittal2024symmetry.

PARAMETER	DESCRIPTION
use_data_augmentation	Whether to use symmetry-based data augmentation.
use_mirror_loss	Whether to use the symmetry-augmentation loss.
data_augmentation_func	Not declared as a field yet (commented out below), so these options reject it; an algorithm that takes one declares its own (HIMPPOAlgorithmOptions.data_augmentation_func). The symmetry data augmentation function, with signature func(env, obs, action) -> (obs, action) where: env (VecEnv): the environment object, used to access its properties. obs (tensordict.TensorDict | None): the observation dictionary; if None, the observation is not used. action (torch.Tensor | None): the action tensor; if None, the action is not used. It returns a tuple of the augmented observation dictionary and action tensors, either of which can be None if its respective input was None.
mirror_loss_coeff	The weight for the symmetry-mirror loss.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

use_data_augmentation class-attribute instance-attribute

use_data_augmentation: bool = False

use_mirror_loss class-attribute instance-attribute

use_mirror_loss: bool = False

mirror_loss_coeff class-attribute instance-attribute

mirror_loss_coeff: float = 0.0

Spatial-softmax CNN

CNN policy head with a spatial-softmax pooling layer.

Spatial softmax (Levine et al. 2016, "End-to-End Training of Deep Visuomotor Policies") replaces the flatten at the end of a conv stack with a soft-argmax over each channel's feature map, returning the expected (x, y) location of every channel — i.e. explicit 2D keypoint coordinates. For a grasp/lift task this hands the policy the object's image-space position directly instead of forcing the MLP trunk to recover it from a flattened feature vector, which is what makes vision manipulation learnable from scratch.

:class:SpatialSoftmaxCNNModel is a drop-in replacement for rsl_rl's CNNModel: point an actor/critic class_name at eden.options.learning.rsl_rl.spatial_softmax:SpatialSoftmaxCNNModel and pass a cnn_cfg with spatial_softmax: True (see eden/tasks/manipulation/lift_box_rgb/config.py). Built on the stock rsl_rl.modules.CNN / rsl_rl.models.{CNNModel,MLPModel} APIs, so it tracks whatever rsl_rl version eden resolves.

Classes

SpatialSoftmax

SpatialSoftmax(height: int, width: int, temperature: float = 1.0)

Bases: Module

Spatial soft-argmax over feature maps.

Given input of shape (B, C, H, W), computes a softmax over each channel's spatial locations and returns the expected (x, y) coordinates, yielding output shape (B, C * 2).

Args: height: Height of the input feature maps. width: Width of the input feature maps. temperature: Temperature for the spatial softmax. Lower values produce sharper distributions.

Source code in eden/options/learning/rsl_rl/spatial_softmax.py

def __init__(self, height: int, width: int, temperature: float = 1.0) -> None:
    super().__init__()
    # Normalised coordinate grids in [-1, 1].
    pos_x, pos_y = torch.meshgrid(
        torch.linspace(-1.0, 1.0, height),
        torch.linspace(-1.0, 1.0, width),
        indexing="ij",
    )
    # Register as buffers so they move with the module's device.
    self.register_buffer("pos_x", pos_x.reshape(1, 1, -1))
    self.register_buffer("pos_y", pos_y.reshape(1, 1, -1))
    self.temperature = temperature

Attributes

temperature instance-attribute

temperature = temperature

Methods:

forward

forward(x: Tensor) -> Tensor

Source code in eden/options/learning/rsl_rl/spatial_softmax.py

def forward(self, x: torch.Tensor) -> torch.Tensor:
    B, C, H, W = x.shape
    features = x.reshape(B, C, -1)
    weights = torch.softmax(features / self.temperature, dim=-1)
    pos_x: torch.Tensor = self.pos_x  # type: ignore[assignment]
    pos_y: torch.Tensor = self.pos_y  # type: ignore[assignment]
    expected_x = (weights * pos_x).sum(dim=-1)
    expected_y = (weights * pos_y).sum(dim=-1)
    return torch.stack([expected_x, expected_y], dim=-1).reshape(B, C * 2)

SpatialSoftmaxCNN

SpatialSoftmaxCNN(input_dim: tuple[int, int], input_channels: int, temperature: float = 1.0, **cnn_kwargs: Any)

Bases: Module

CNN encoder with spatial-softmax pooling.

Wraps rsl_rl.modules.CNN (created with global_pool="none" and flatten=False) followed by :class:SpatialSoftmax. Exposes the same interface as CNN: :attr:output_dim (int) and :attr:output_channels (None, signalling flattened output).

Args: input_dim: (H, W) of the input images. input_channels: Number of input channels. temperature: Temperature for the spatial softmax. **cnn_kwargs: Remaining keyword arguments forwarded to rsl_rl.modules.CNN.

Source code in eden/options/learning/rsl_rl/spatial_softmax.py

def __init__(
    self,
    input_dim: tuple[int, int],
    input_channels: int,
    temperature: float = 1.0,
    **cnn_kwargs: Any,
) -> None:
    super().__init__()
    # Spatial softmax replaces pooling/flatten.
    cnn_kwargs.pop("global_pool", None)
    cnn_kwargs.pop("flatten", None)
    self.cnn = CNN(
        input_dim=input_dim,
        input_channels=input_channels,
        global_pool="none",
        flatten=False,
        **cnn_kwargs,
    )
    # cnn.output_dim is (H, W) when flatten=False.
    out_h, out_w = self.cnn.output_dim  # type: ignore[misc]
    num_channels: int = self.cnn.output_channels  # type: ignore[assignment]
    self.spatial_softmax = SpatialSoftmax(out_h, out_w, temperature)
    self._output_dim = num_channels * 2

Attributes

cnn instance-attribute

cnn = CNN(input_dim=input_dim, input_channels=input_channels, global_pool='none', flatten=False, **cnn_kwargs)

spatial_softmax instance-attribute

spatial_softmax = SpatialSoftmax(out_h, out_w, temperature)

output_dim property

output_dim: int

Total flattened output dimension (C * 2).

output_channels property

output_channels: None

Always None (output is flattened keypoints).

Methods:

forward

forward(x: Tensor) -> Tensor

Source code in eden/options/learning/rsl_rl/spatial_softmax.py

def forward(self, x: torch.Tensor) -> torch.Tensor:
    features = self.cnn(x)
    return self.spatial_softmax(features)

SpatialSoftmaxCNNModel

SpatialSoftmaxCNNModel(obs: TensorDict, obs_groups: dict[str, list[str]], obs_set: str, output_dim: int, cnn_cfg: dict[str, dict] | dict[str, Any], cnns: ModuleDict | None = None, hidden_dims: tuple[int] | list[int] = [256, 256, 256], activation: str = 'elu', obs_normalization: bool = False, distribution_cfg: dict[str, Any] | None = None)

Bases: CNNModel

CNN model that uses spatial-softmax pooling.

Drop-in replacement for rsl_rl.models.CNNModel. The only difference is that each CNN encoder is wrapped with :class:SpatialSoftmaxCNN instead of a plain CNN. The spatial_softmax / spatial_softmax_temperature keys are stripped from cnn_cfg before the rest is forwarded to the CNN.

Source code in eden/options/learning/rsl_rl/spatial_softmax.py

def __init__(
    self,
    obs: TensorDict,
    obs_groups: dict[str, list[str]],
    obs_set: str,
    output_dim: int,
    cnn_cfg: dict[str, dict] | dict[str, Any],
    cnns: nn.ModuleDict | None = None,
    hidden_dims: tuple[int] | list[int] = [256, 256, 256],  # noqa: B006
    activation: str = "elu",
    obs_normalization: bool = False,
    distribution_cfg: dict[str, Any] | None = None,
) -> None:
    # Separate 1D / 2D observation groups (sets self.obs_groups_2d,
    # obs_dims_2d, obs_channels_2d; returns 1D info for MLPModel).
    self._get_obs_dim(obs, obs_groups, obs_set)

    if cnns is not None:
        if set(cnns.keys()) != set(self.obs_groups_2d):
            raise ValueError("The 2D observations must be identical for all models sharing CNN encoders.")
        print("Sharing CNN encoders between models, the CNN configurations of the receiving model are ignored.")
        _cnns = cnns
    else:
        # Expand a single flat config to per-group configs.
        if not all(isinstance(v, dict) for v in cnn_cfg.values()):
            cnn_cfg = {group: cnn_cfg for group in self.obs_groups_2d}
        assert len(cnn_cfg) == len(self.obs_groups_2d), (
            "The number of CNN configurations must match the number of 2D observation groups."
        )
        _cnns = {}
        for idx, obs_group in enumerate(self.obs_groups_2d):
            group_cfg = dict(cnn_cfg[obs_group])
            group_cfg.pop("spatial_softmax", None)
            temperature = group_cfg.pop("spatial_softmax_temperature", 1.0)
            _cnns[obs_group] = SpatialSoftmaxCNN(
                input_dim=self.obs_dims_2d[idx],
                input_channels=self.obs_channels_2d[idx],
                temperature=temperature,
                **group_cfg,
            )

    self.cnn_latent_dim = 0
    for cnn in _cnns.values():
        if cnn.output_channels is not None:
            raise ValueError("The output of the CNN must be flattened before passing it to the MLP.")
        self.cnn_latent_dim += int(cnn.output_dim)  # type: ignore[arg-type]

    MLPModel.__init__(
        self,
        obs=obs,
        obs_groups=obs_groups,
        obs_set=obs_set,
        output_dim=output_dim,
        hidden_dims=hidden_dims,
        activation=activation,
        obs_normalization=obs_normalization,
        distribution_cfg=distribution_cfg,
    )

    if isinstance(_cnns, nn.ModuleDict):
        self.cnns = _cnns
    else:
        self.cnns = nn.ModuleDict(_cnns)

Attributes

cnn_latent_dim instance-attribute

cnn_latent_dim = 0

cnns instance-attribute

cnns = _cnns

Methods:

Version translation

Helpers that detect the installed rsl-rl-lib version and translate the typed options into the runner dict it expects.

Translation functions between rsl_rl config versions.

These functions convert the canonical v5-style runner dict (separate actor / critic with distribution_cfg) into the formats expected by older rsl_rl versions.

Typical usage::

from eden.options.learning.rsl_rl.translate import translate_runner_dict

runner_dict = translate_runner_dict(config.runner_options.dict())  # auto-detect
runner = OnPolicyRunner(env, runner_dict, ...)

Functions:

detect_rsl_rl_major_version cached

detect_rsl_rl_major_version() -> int

Return the major version of the installed rsl-rl-lib package.

Falls back to 3 if the package metadata is unavailable (e.g. editable install from source without version).

Source code in eden/options/learning/rsl_rl/translate.py

@functools.lru_cache(maxsize=1)
def detect_rsl_rl_major_version() -> int:
    """Return the major version of the installed ``rsl-rl-lib`` package.

    Falls back to ``3`` if the package metadata is unavailable (e.g. editable
    install from source without version).
    """
    try:
        from importlib.metadata import version

        return int(version("rsl-rl-lib").split(".")[0])
    except Exception:
        return 3

detect_rsl_rl_version cached

detect_rsl_rl_version() -> tuple[int, ...]

Return the installed rsl-rl-lib release as a tuple ((5, 5, 1)), for checks finer than the major.

Falls back to (3,), as :func:detect_rsl_rl_major_version does, if the package metadata is unavailable.

Source code in eden/options/learning/rsl_rl/translate.py

@functools.lru_cache(maxsize=1)
def detect_rsl_rl_version() -> tuple[int, ...]:
    """Return the installed ``rsl-rl-lib`` release as a tuple (``(5, 5, 1)``), for checks finer than the major.

    Falls back to ``(3,)``, as :func:`detect_rsl_rl_major_version` does, if the package metadata is unavailable.
    """
    try:
        from importlib.metadata import version

        from packaging.version import Version

        return Version(version("rsl-rl-lib")).release
    except Exception:
        return (3,)

require_rsl_rl_version

require_rsl_rl_version(minimum: tuple[int, ...], module: str, needs: str) -> None

Raise an ImportError that names the upgrade when the installed rsl-rl-lib is older than minimum.

Called at the top of a module before its rsl_rl imports, so an older install reads "upgrade to X" rather than cannot import name ... from inside rsl_rl.

PARAMETER	DESCRIPTION
minimum	The oldest release the module runs on. TYPE: tuple[int, ...]
module	The module's dotted name, for the message. TYPE: str
needs	What it uses that older releases lack, for the message. TYPE: str

Source code in eden/options/learning/rsl_rl/translate.py

def require_rsl_rl_version(minimum: tuple[int, ...], module: str, needs: str) -> None:
    """Raise an ``ImportError`` that names the upgrade when the installed ``rsl-rl-lib`` is older than ``minimum``.

    Called at the top of a module before its ``rsl_rl`` imports, so an older install reads "upgrade to X" rather than
    ``cannot import name ...`` from inside rsl_rl.

    Parameters
    ----------
    minimum : tuple[int, ...]
        The oldest release the module runs on.
    module : str
        The module's dotted name, for the message.
    needs : str
        What it uses that older releases lack, for the message.
    """
    installed = detect_rsl_rl_version()
    if installed < minimum:
        wanted = ".".join(map(str, minimum))
        raise ImportError(
            f"{module} needs rsl-rl-lib >= {wanted} ({needs}); the installed version is "
            f"{'.'.join(map(str, installed))}. Upgrade with `uv pip install 'rsl-rl-lib>={wanted}'`."
        )

translate_runner_dict

translate_runner_dict(runner_dict: dict, algorithm: Any = None) -> dict

Translate runner_dict to whatever the installed rsl_rl version expects.

Auto-detects the installed rsl_rl version and translates from the canonical v5 format to whatever that version expects.

PARAMETER	DESCRIPTION
runner_dict	A runner configuration dict (v5-style with actor / critic keys). Not modified in-place — a deep copy is returned. TYPE: dict
algorithm	The runner's algorithm options. When they define adapt_runner_dict(runner_dict, major), it runs on the translated dict, so an algorithm outside rsl_rl (an eden.learning.wrappers.rsl_rl one) adapts its own entries for the installed major — the generic translation only knows rsl_rl's own models. TYPE: ConfigurableOptions DEFAULT: None

RETURNS	DESCRIPTION
dict	A runner dict compatible with the installed rsl_rl version.

Source code in eden/options/learning/rsl_rl/translate.py

def translate_runner_dict(runner_dict: dict, algorithm: Any = None) -> dict:
    """Translate *runner_dict* to whatever the installed rsl_rl version expects.

    Auto-detects the installed rsl_rl version and translates from the canonical
    v5 format to whatever that version expects.

    Parameters
    ----------
    runner_dict : dict
        A runner configuration dict (v5-style with ``actor`` / ``critic`` keys).
        **Not modified in-place** — a deep copy is returned.
    algorithm : ConfigurableOptions, optional
        The runner's algorithm options. When they define ``adapt_runner_dict(runner_dict, major)``,
        it runs on the translated dict, so an algorithm outside rsl_rl (an
        ``eden.learning.wrappers.rsl_rl`` one) adapts its own entries for the installed major
        — the generic translation only knows rsl_rl's own models.

    Returns
    -------
    dict
        A runner dict compatible with the installed rsl_rl version.
    """
    major = detect_rsl_rl_major_version()
    if major < 3:
        raise ValueError(
            f"Detected rsl_rl version {major}.X is not supported. Only versions 3, 4, and 5 are supported."
        )
    elif major == 3:
        en.logger.info("Detected rsl_rl version 3.X. Translating runner dict to v3 format.")
        out = translate_runner_dict_to_v3(runner_dict)
    elif major == 4:
        en.logger.info("Detected rsl_rl version 4.X. Translating runner dict to v4 format.")
        out = translate_runner_dict_to_v4(runner_dict)
    else:  # v5+
        out = copy.deepcopy(runner_dict)
    adapt = getattr(algorithm, "adapt_runner_dict", None)  # an optional hook, not a required field
    return adapt(out, major) if adapt is not None else out

translate_runner_dict_to_v3

translate_runner_dict_to_v3(runner_dict: dict) -> dict

Convert a v5-style runner dict to the v3.2 format.

v3.2 expects: - A single "policy" dict with combined actor/critic fields (actor_hidden_dims, critic_hidden_dims, init_noise_std, …). - The actor observation set keyed as "policy" in obs_groups. - No share_cnn_encoders or check_for_nan in the algorithm.

PARAMETER	DESCRIPTION
runner_dict	A runner configuration dict produced by RslRlOnPolicyRunnerOptions.dict(). Not modified in-place — a deep copy is returned. TYPE: dict

RETURNS	DESCRIPTION
dict	A v3.2-compatible runner dict.

Source code in eden/options/learning/rsl_rl/translate.py

def translate_runner_dict_to_v3(runner_dict: dict) -> dict:
    """Convert a v5-style runner dict to the v3.2 format.

    v3.2 expects:
    - A single ``"policy"`` dict with combined actor/critic fields
      (``actor_hidden_dims``, ``critic_hidden_dims``, ``init_noise_std``, …).
    - The actor observation set keyed as ``"policy"`` in ``obs_groups``.
    - No ``share_cnn_encoders`` or ``check_for_nan`` in the algorithm.

    Parameters
    ----------
    runner_dict : dict
        A runner configuration dict produced by ``RslRlOnPolicyRunnerOptions.dict()``.
        **Not modified in-place** — a deep copy is returned.

    Returns
    -------
    dict
        A v3.2-compatible runner dict.
    """
    out = copy.deepcopy(runner_dict)

    # If already in v3.2 format, return as-is
    if "policy" in out and "actor" not in out:
        return out

    actor = out.pop("actor", {})
    critic = out.pop("critic", {})

    # --- build combined policy dict ------------------------------------------
    dist_cfg = actor.pop("distribution_cfg", None)
    class_name = _model_class_to_v3(actor.get("class_name", "MLPModel"))

    policy: dict[str, Any] = {
        "class_name": class_name,
        "actor_hidden_dims": actor.get("hidden_dims", [256, 256, 256]),
        "critic_hidden_dims": critic.get("hidden_dims", [256, 256, 256]),
        "activation": actor.get("activation", "elu"),
        "actor_obs_normalization": actor.get("obs_normalization", False),
        "critic_obs_normalization": critic.get("obs_normalization", False),
    }

    # Distribution → noise params
    if dist_cfg is not None:
        policy["init_noise_std"] = dist_cfg.get("init_std", 1.0)
        policy["noise_std_type"] = dist_cfg.get("std_type", "scalar")

    # Forward RNN fields
    for key in ("rnn_type", "rnn_hidden_dim", "rnn_num_layers"):
        if key in actor:
            policy[key] = actor[key]

    # CNN: v3's ``ActorCriticCNN`` takes separate ``actor_cnn_cfg`` / ``critic_cnn_cfg``
    # (one conv stack per 2D obs group per side), not the v5 single ``cnn_cfg`` — a
    # bare ``cnn_cfg`` lands in its ``**kwargs`` and is silently ignored, leaving
    # the model with no CNN for its image obs.
    if class_name == "ActorCriticCNN":
        if actor.get("cnn_cfg") is not None:
            policy["actor_cnn_cfg"] = actor["cnn_cfg"]
        if critic.get("cnn_cfg") is not None:
            policy["critic_cnn_cfg"] = critic["cnn_cfg"]
    elif actor.get("cnn_cfg") is not None:
        policy["cnn_cfg"] = actor["cnn_cfg"]

    # --- obs_groups: rename "actor" → "policy" --------------------------------
    obs_groups = out.get("obs_groups", {})
    if "actor" in obs_groups and "policy" not in obs_groups:
        obs_groups["policy"] = obs_groups.pop("actor")
    out["obs_groups"] = obs_groups

    # --- algorithm: strip v5-only fields -------------------------------------
    alg = out.get("algorithm", {})
    alg.pop("share_cnn_encoders", None)
    alg.pop("check_for_nan", None)

    # --- runner: strip v5-only fields ----------------------------------------
    out.pop("check_for_nan", None)

    # --- runner: add deprecated field for compat -----------------------------
    out.setdefault("empirical_normalization", None)

    out["policy"] = policy
    return out

translate_runner_dict_to_v4

translate_runner_dict_to_v4(runner_dict: dict) -> dict

Convert a v5-style runner dict to the v4 format.

v4 uses separated model dicts (like v5) — actor / critic for PPO, student / teacher for distillation — but a model takes stochastic, init_noise_std, noise_std_type, state_dependent_std instead of distribution_cfg. Both v4 runners splat each model dict straight into MLPModel.__init__, so an untranslated distribution_cfg is a TypeError there.

PARAMETER	DESCRIPTION
runner_dict	A runner configuration dict produced by RslRlOnPolicyRunnerOptions.dict() or RslRlDistillationRunnerOptions.dict(). Not modified in-place — a deep copy is returned. TYPE: dict

RETURNS	DESCRIPTION
dict	A v4-compatible runner dict.

Source code in eden/options/learning/rsl_rl/translate.py

def translate_runner_dict_to_v4(runner_dict: dict) -> dict:
    """Convert a v5-style runner dict to the v4 format.

    v4 uses separated model dicts (like v5) — ``actor`` / ``critic`` for PPO,
    ``student`` / ``teacher`` for distillation — but a model takes
    ``stochastic``, ``init_noise_std``, ``noise_std_type``,
    ``state_dependent_std`` instead of ``distribution_cfg``. Both v4 runners
    splat each model dict straight into ``MLPModel.__init__``, so an
    untranslated ``distribution_cfg`` is a ``TypeError`` there.

    Parameters
    ----------
    runner_dict : dict
        A runner configuration dict produced by ``RslRlOnPolicyRunnerOptions.dict()``
        or ``RslRlDistillationRunnerOptions.dict()``.
        **Not modified in-place** — a deep copy is returned.

    Returns
    -------
    dict
        A v4-compatible runner dict.
    """
    out = copy.deepcopy(runner_dict)

    # --- convert distribution_cfg to v4 noise params -------------------------
    # Only the heads the dict has: a distillation dict gains no ``actor``.
    for head in ("actor", "student", "teacher"):
        if isinstance(out.get(head), dict):
            _model_to_v4(out[head])

    # --- critic: strip distribution_cfg if present (shouldn't be, but safe) --
    if isinstance(out.get("critic"), dict):
        out["critic"].pop("distribution_cfg", None)

    # --- algorithm: strip v5-only fields -------------------------------------
    alg = out.get("algorithm", {})
    alg.pop("check_for_nan", None)

    # --- runner: strip v5-only fields ----------------------------------------
    out.pop("check_for_nan", None)

    return out

rl_games

rl_games network and PPO runner configuration options.

Attributes

Classes

RlGamesNetworkOptions

RlGamesNetworkOptions(**data)

Bases: ConfigurableOptions

Network architecture configuration for rl_games.

PARAMETER	DESCRIPTION
name	The network builder name.
separate	Whether actor and critic have separate network trunks.
units	Hidden layer sizes for the MLP.
activation	Activation function name.
mu_activation	Activation for the mean output head. "None" maps to identity.
sigma_activation	Activation for the sigma output head. "None" maps to identity.
fixed_sigma	Whether sigma is a fixed (learned) parameter or state-dependent.
init_sigma_val	Initial value for sigma (const initializer).
mu_init_name	Initializer name for mu weights.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

name class-attribute instance-attribute

name: str = 'actor_critic'

separate class-attribute instance-attribute

separate: bool = True

units class-attribute instance-attribute

units: list[int] = [512, 256, 128]

activation class-attribute instance-attribute

activation: str = 'elu'

mu_activation class-attribute instance-attribute

mu_activation: str = 'None'

sigma_activation class-attribute instance-attribute

sigma_activation: str = 'None'

fixed_sigma class-attribute instance-attribute

fixed_sigma: bool = True

init_sigma_val class-attribute instance-attribute

init_sigma_val: float = 0.0

mu_init_name class-attribute instance-attribute

mu_init_name: str = 'default'

RlGamesPpoRunnerOptions

RlGamesPpoRunnerOptions(**data)

Bases: ConfigurableOptions

PPO runner configuration for rl_games.

Mirrors the params dict expected by :class:rl_games.torch_runner.Runner. :func:~eden.options.learning.rl_games.translate.translate_runner_dict translates these options into the nested dict that Runner.load() consumes.

PARAMETER	DESCRIPTION
seed	Random seed. Non-negative; see eden.utils.random.validate_seed.
experiment_name	Name for the experiment (used in log/checkpoint dirs).
device	Torch device string.
max_epochs	Maximum number of training epochs.
save_frequency	Save checkpoint every N epochs. 0 disables periodic saving.
keep_last_checkpoints	How many of the run's newest periodic last_<name>_ep_<epoch>_rew_<reward>.pth checkpoints to keep on disk; each save deletes the older ones. Defaults to 1, the newest; None keeps every checkpoint. The best-reward checkpoint <name>.pth is never deleted.
horizon_length	Number of environment steps per rollout.
mini_epochs	Number of PPO optimisation passes per rollout.
minibatch_size	Minibatch size. If None, defaults to horizon_length * num_envs (full-batch), times len(agents) when agents is set.
learning_rate	Initial learning rate.
lr_schedule	Learning-rate schedule ("adaptive", "linear", or "None").
kl_threshold	Target KL divergence for adaptive LR.
gamma	Discount factor.
tau	GAE lambda.
e_clip	PPO clipping parameter.
clip_value	Whether to clip value-function loss.
entropy_coef	Entropy bonus coefficient.
critic_coef	Value-loss coefficient.
grad_norm	Maximum gradient norm for clipping.
truncate_grads	Whether to clip gradients.
normalize_input	Whether to normalize observations with running statistics.
normalize_value	Whether to normalize value targets.
normalize_advantage	Whether to normalize advantages.
value_bootstrap	Whether to bootstrap value on time-outs.
reward_scale	Multiplicative reward scale applied by the reward shaper.
clip_actions	Symmetric action clipping bound: a finite positive bound, or None to disable clipping.
bounds_loss_coef	Coefficient for the action-bounds regularisation loss.
obs_groups	Mapping from "actor" / "critic" to observation group names.
agents	Names the agents of a multi-agent env, in order. Each name must be an observation group, a reward group and an action term of the env, and the action terms must be declared in this same order. rl_games then trains one shared policy over num_envs * len(agents) rows (env_info["agents"] = len(agents)), each fed its agent's observation group, action term and group reward. Empty (the default) is the single-agent wrapper. Mutually exclusive with obs_groups: a multi-agent run has no central-value critic, and each agent's observation is its own group.
network	Network architecture.
torch_compile	torch.compile configuration passed to the runner.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

seed class-attribute instance-attribute

seed: Seed = 7100

experiment_name class-attribute instance-attribute

experiment_name: str = ''

device class-attribute instance-attribute

device: DeviceStr = 'auto'

max_epochs class-attribute instance-attribute

max_epochs: int = 30000

save_frequency class-attribute instance-attribute

save_frequency: int = 500

keep_last_checkpoints class-attribute instance-attribute

keep_last_checkpoints: int | None = 1

horizon_length class-attribute instance-attribute

horizon_length: int = 24

mini_epochs class-attribute instance-attribute

mini_epochs: int = 5

minibatch_size class-attribute instance-attribute

minibatch_size: int | None = None

learning_rate class-attribute instance-attribute

learning_rate: float = 0.001

lr_schedule class-attribute instance-attribute

lr_schedule: Literal['adaptive', 'linear', 'None'] = 'adaptive'

kl_threshold class-attribute instance-attribute

kl_threshold: float = 0.01

gamma class-attribute instance-attribute

gamma: float = 0.99

tau class-attribute instance-attribute

tau: float = 0.95

e_clip class-attribute instance-attribute

e_clip: float = 0.2

clip_value class-attribute instance-attribute

clip_value: bool = True

entropy_coef class-attribute instance-attribute

entropy_coef: float = 0.01

critic_coef class-attribute instance-attribute

critic_coef: float = 1.0

grad_norm class-attribute instance-attribute

grad_norm: float = 1.0

truncate_grads class-attribute instance-attribute

truncate_grads: bool = True

normalize_input class-attribute instance-attribute

normalize_input: bool = False

normalize_value class-attribute instance-attribute

normalize_value: bool = False

normalize_advantage class-attribute instance-attribute

normalize_advantage: bool = True

value_bootstrap class-attribute instance-attribute

value_bootstrap: bool = True

reward_scale class-attribute instance-attribute

reward_scale: float = 1.0

clip_actions class-attribute instance-attribute

clip_actions: ClipActions | None = None

bounds_loss_coef class-attribute instance-attribute

bounds_loss_coef: float = 0.0

obs_groups class-attribute instance-attribute

obs_groups: dict[str, list[str]] = {}

agents class-attribute instance-attribute

agents: list[str] = []

network class-attribute instance-attribute

network: RlGamesNetworkOptions = RlGamesNetworkOptions()

torch_compile class-attribute instance-attribute

torch_compile: bool | str | dict | None = False

Functions:

check_agents_options

check_agents_options(agents: list[str], obs_groups: dict[str, list[str]]) -> None

Raise ValueError if agents repeats a name or is set together with obs_groups.

Shared by the options validator and :class:~eden.envs.wrappers.rl_games_env.RlGamesVecEnvWrapper, which re-checks because a field assigned after construction skips the validator.

Source code in eden/options/learning/rl_games/rl.py

def check_agents_options(agents: list[str], obs_groups: dict[str, list[str]]) -> None:
    """Raise ``ValueError`` if ``agents`` repeats a name or is set together with ``obs_groups``.

    Shared by the options validator and :class:`~eden.envs.wrappers.rl_games_env.RlGamesVecEnvWrapper`,
    which re-checks because a field assigned after construction skips the validator.
    """
    if not agents:
        return
    if obs_groups:
        raise ValueError(
            f"RlGamesPpoRunnerOptions sets both `agents={agents!r}` and `obs_groups={obs_groups!r}`; "
            "they are mutually exclusive.\n"
            "In multi-agent mode each agent's observation is the observation group named after it, "
            "and there is no central-value critic to give `obs_groups['critic']` to.\n"
            "Fix: drop `obs_groups` (set it to `{}`) to train the agents, or drop `agents` (set it to `[]`) "
            "for a single-agent run."
        )
    repeated = sorted({name for name in agents if agents.count(name) > 1})
    if repeated:
        raise ValueError(
            f"RlGamesPpoRunnerOptions.agents names {repeated!r} more than once: {agents!r}.\n"
            "Each agent owns one observation group, one reward group and one action term.\n"
            "Fix: list every agent exactly once, in the action manager's term order."
        )

Translation from Eden rl_games options to the nested dict rl_games expects.

Typical usage::

from eden.options.learning.rl_games.translate import translate_runner_dict

runner_dict = translate_runner_dict(config.runner_options.dict(), num_envs=env.num_envs)

from rl_games.torch_runner import Runner
runner = Runner()
runner.load(runner_dict)

Functions:

translate_runner_dict

translate_runner_dict(runner_dict: dict[str, Any], *, num_envs: int) -> dict[str, Any]

Convert a flat Eden options dict into the nested params dict rl_games expects.

The output matches what rl_games.torch_runner.Runner.load() expects.

PARAMETER	DESCRIPTION
runner_dict	A dict produced by RlGamesPpoRunnerOptions.dict(). TYPE: dict[str, Any]
num_envs	Number of parallel environments (injected by the wrapper). TYPE: int

RETURNS	DESCRIPTION
dict	The full {"params": {...}} dict ready for Runner.load().

Source code in eden/options/learning/rl_games/translate.py

def translate_runner_dict(runner_dict: dict[str, Any], *, num_envs: int) -> dict[str, Any]:
    """Convert a flat Eden options dict into the nested ``params`` dict rl_games expects.

    The output matches what ``rl_games.torch_runner.Runner.load()`` expects.

    Parameters
    ----------
    runner_dict:
        A dict produced by ``RlGamesPpoRunnerOptions.dict()``.
    num_envs:
        Number of parallel environments (injected by the wrapper).

    Returns
    -------
    dict
        The full ``{"params": {...}}`` dict ready for ``Runner.load()``.
    """
    from eden.utils.torch import resolve_device

    d = runner_dict
    net = d.get("network", {})

    minibatch_size = d.get("minibatch_size")
    if minibatch_size is None:
        # Full batch. A multi-agent env (`agents`) gives rl_games `num_envs * len(agents)` rows a step;
        # `agents` itself is a wrapper concern rl_games never reads, so it is not forwarded.
        minibatch_size = d.get("horizon_length", 24) * num_envs * max(len(d.get("agents") or ()), 1)

    lr_schedule = d.get("lr_schedule", "adaptive")
    if lr_schedule == "None":
        lr_schedule = None

    return {
        "params": {
            "seed": d.get("seed", 7100),
            "algo": {"name": "a2c_continuous"},
            "model": {"name": "continuous_a2c_logstd"},
            "network": {
                "name": net.get("name", "actor_critic"),
                "separate": net.get("separate", True),
                "space": {
                    "continuous": {
                        "mu_activation": net.get("mu_activation", "None"),
                        "sigma_activation": net.get("sigma_activation", "None"),
                        "mu_init": {"name": net.get("mu_init_name", "default")},
                        "sigma_init": {
                            "name": "const_initializer",
                            "val": net.get("init_sigma_val", 0.0),
                        },
                        "fixed_sigma": net.get("fixed_sigma", True),
                    },
                },
                "mlp": {
                    "units": list(net.get("units", [512, 256, 128])),
                    "activation": net.get("activation", "elu"),
                    "initializer": {"name": "default"},
                },
            },
            "config": {
                "name": d.get("experiment_name", ""),
                "env_name": "eden",
                "device": resolve_device(d.get("device", "auto")),
                "num_actors": num_envs,
                "horizon_length": d.get("horizon_length", 24),
                "minibatch_size": minibatch_size,
                "mini_epochs": d.get("mini_epochs", 5),
                "learning_rate": d.get("learning_rate", 1e-3),
                "lr_schedule": lr_schedule,
                "kl_threshold": d.get("kl_threshold", 0.01),
                "gamma": d.get("gamma", 0.99),
                "tau": d.get("tau", 0.95),
                "e_clip": d.get("e_clip", 0.2),
                "clip_value": d.get("clip_value", True),
                "entropy_coef": d.get("entropy_coef", 0.01),
                "critic_coef": d.get("critic_coef", 1.0),
                "grad_norm": d.get("grad_norm", 1.0),
                "truncate_grads": d.get("truncate_grads", True),
                "normalize_input": d.get("normalize_input", False),
                "normalize_value": d.get("normalize_value", False),
                "normalize_advantage": d.get("normalize_advantage", True),
                "value_bootstrap": d.get("value_bootstrap", True),
                "reward_shaper": {"scale_value": d.get("reward_scale", 1.0)},
                "bounds_loss_coef": d.get("bounds_loss_coef", 0.0),
                # Disable rl_games' built-in action rescaling — Eden
                # handles action scaling internally via controllers.
                "clip_actions": False,
                "max_epochs": d.get("max_epochs", 30000),
                "save_frequency": d.get("save_frequency", 500),
                "torch_compile": d.get("torch_compile", True),
                "print_stats": True,
            },
        },
    }

Action clipping

The clip_actions bound that a runner's options and every env wrapper accept.

The action bound a runner's options and every env wrapper accept: clip_actions.

Attributes

ClipActions module-attribute

ClipActions = Annotated[float, AfterValidator(validate_clip_actions)]

Functions:

validate_clip_actions

validate_clip_actions(clip_actions: float, field: str = 'clip_actions') -> float

Return clip_actions if it is a finite positive bound, raising otherwise.

Actions are clamped to [-clip_actions, clip_actions]: zero would clamp every action to zero, a negative bound inverts the interval, and an infinite or NaN one bounds nothing while promising a bound to a library that reads the action space. field is where the caller set it, for the message.

Source code in eden/options/learning/clip_actions.py

def validate_clip_actions(clip_actions: float, field: str = "clip_actions") -> float:
    """Return ``clip_actions`` if it is a finite positive bound, raising otherwise.

    Actions are clamped to ``[-clip_actions, clip_actions]``: zero would clamp every action to zero, a
    negative bound inverts the interval, and an infinite or NaN one bounds nothing while promising a
    bound to a library that reads the action space. ``field`` is where the caller set it, for the message.
    """
    if not (0 < clip_actions < math.inf):
        raise ValueError(
            f"{field}={clip_actions!r} must be a finite positive bound, or None to leave actions unclamped."
        )
    return clip_actions

See also

• Training runs guide
• Learning: algorithms and export
• Other RL libraries