Retargeting

Retargeting maps motion captured on one morphology (human keypoints, SMPL-family bodies, MANO hands, interaction meshes) onto an Eden robot. The four solvers are RetargetingBase subclasses configured by a RetargeterOptions, and the FlashDexRetarget pipeline is a set of functions with its own options; each imports its backend lazily. The retargeting guide covers eden retarget and the body backends; the dex retargeting guide covers hands.

from eden.extensions.retargeting.base import RetargetingBase
from eden.options.extensions.retargeting import RetargeterOptions

Backend	Maps	CLI
DexRetargeter	MANO hand keypoints → a dexterous robot hand	eden retarget dex
OmniRetargeter	human motion with an interaction mesh → a humanoid (Laplacian deformation)	eden retarget omniretarget
OTRetargeter	a human surface and objects → a robot and objects (optimal transport)	eden retarget otretarget
SomaRetargeter	SMPL / SMPL-H / SMPL-X / SOMA → a floating-base humanoid (Unitree G1, LimX Oli), through the eden-soma-retargeter library	eden retarget soma
FlashDexRetarget	MANO hand-object demos → a bimanual robot-hand motion.pt	eden retarget flashdex

Base class and options

RetargetingBase

RetargetingBase(env: EnvBase, options: RetargeterOptions)

Bases: ConfigurableMixin[RetargeterOptions]

Abstract base class for retargeting backends.

Resolves entity_name against env.entities at construction (raising on an unknown name); subclasses implement :meth:retarget_motion.

Source code in eden/extensions/retargeting/base.py

def __init__(self, env: EnvBase, options: RetargeterOptions) -> None:
    super().__init__(options)
    self.env = env
    self._entity = resolve_named_entity(
        env.entities, self.entity_name, hint="RetargeterOptions.entity_name names it; set it to one of those."
    )

Attributes

entity_name class-attribute instance-attribute

entity_name: str = ''

env instance-attribute

env = env

entity property

entity: Entity

Methods:

retarget_motion abstractmethod

retarget_motion()

Source code in eden/extensions/retargeting/base.py

@abstractmethod
def retarget_motion(self):
    raise NotImplementedError

RetargeterOptions

RetargeterOptions(**data)

Bases: DynamicConfigurableOptions

Options for retargeting extensions.

PARAMETER	DESCRIPTION
entity_name	The name of the entity to perform retargeting for. TYPE: str

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

entity_name class-attribute instance-attribute

entity_name: str = 'robot'

Shared constants for retargeting.

Attributes

JOINTS_MAP_UNITREE_G1 module-attribute

JOINTS_MAP_UNITREE_G1 = {'smplh': {'Pelvis': 'pelvis', 'L_Hip': 'left_hip_pitch_link', 'R_Hip': 'right_hip_pitch_link', 'L_Knee': 'left_knee_link', 'R_Knee': 'right_knee_link', 'L_Shoulder': 'left_shoulder_roll_link', 'R_Shoulder': 'right_shoulder_roll_link', 'L_Elbow': 'left_elbow_link', 'R_Elbow': 'right_elbow_link', 'L_Ankle': 'left_ankle_pitch_link', 'R_Ankle': 'right_ankle_pitch_link', 'L_Toe': 'left_ankle_roll_link', 'R_Toe': 'right_ankle_roll_link', 'L_Wrist': 'left_wrist_yaw_link', 'R_Wrist': 'right_wrist_yaw_link'}, 'smplx': {'Pelvis': 'pelvis', 'L_Hip': 'left_hip_pitch_link', 'R_Hip': 'right_hip_pitch_link', 'L_Knee': 'left_knee_link', 'R_Knee': 'right_knee_link', 'L_Shoulder': 'left_shoulder_roll_link', 'R_Shoulder': 'right_shoulder_roll_link', 'L_Elbow': 'left_elbow_link', 'R_Elbow': 'right_elbow_link', 'L_Ankle': 'left_ankle_pitch_link', 'R_Ankle': 'right_ankle_pitch_link', 'L_Foot': 'left_ankle_roll_link', 'R_Foot': 'right_ankle_roll_link', 'L_Wrist': 'left_wrist_yaw_link', 'R_Wrist': 'right_wrist_yaw_link'}, 'mocap': {'Spine1': 'pelvis', 'LeftUpLeg': 'left_hip_pitch_link', 'LeftLeg': 'left_knee_link', 'LeftToeBase': 'left_ankle_roll_link', 'RightUpLeg': 'right_hip_pitch_link', 'RightLeg': 'right_knee_link', 'RightToeBase': 'right_ankle_roll_link', 'LeftArm': 'left_shoulder_roll_link', 'LeftForeArm': 'left_elbow_link', 'LeftHandMiddle3': 'left_sphere_hand_link', 'RightArm': 'right_shoulder_roll_link', 'RightForeArm': 'right_elbow_link', 'RightHandMiddle3': 'right_sphere_hand_link', 'LeftFoot': 'left_ankle_pitch_link', 'RightFoot': 'right_ankle_pitch_link'}, 'lafan1': {'Spine1': 'pelvis', 'LeftUpLeg': 'left_hip_pitch_link', 'RightUpLeg': 'right_hip_pitch_link', 'LeftLeg': 'left_knee_link', 'RightLeg': 'right_knee_link', 'LeftArm': 'left_shoulder_roll_link', 'RightArm': 'right_shoulder_roll_link', 'LeftForeArm': 'left_elbow_link', 'RightForeArm': 'right_elbow_link', 'LeftFoot': 'left_ankle_pitch_link', 'RightFoot': 'right_ankle_pitch_link', 'LeftToeBase': 'left_ankle_roll_link', 'RightToeBase': 'right_ankle_roll_link', 'LeftHand': 'left_wrist_yaw_link', 'RightHand': 'right_wrist_yaw_link'}}

JOINTS_MAP_BOOSTER_T1 module-attribute

JOINTS_MAP_BOOSTER_T1 = {'smplh': {'Pelvis': 'Trunk', 'L_Hip': 'Hip_Pitch_Left', 'R_Hip': 'Hip_Pitch_Right', 'L_Knee': 'Shank_Left', 'R_Knee': 'Shank_Right', 'L_Shoulder': 'AL1', 'R_Shoulder': 'AR1', 'L_Elbow': 'left_hand_link', 'R_Elbow': 'right_hand_link', 'L_Ankle': 'Ankle_Cross_Left', 'R_Ankle': 'Ankle_Cross_Right', 'L_Toe': 'left_foot_sphere_5_link', 'R_Toe': 'right_foot_sphere_5_link', 'L_Wrist': 'left_wrist_yaw_link', 'R_Wrist': 'right_wrist_yaw_link'}, 'mocap': {'Spine1': 'Trunk', 'LeftUpLeg': 'Hip_Pitch_Left', 'LeftLeg': 'Shank_Left', 'LeftToeBase': 'left_foot_sphere_5_link', 'RightUpLeg': 'Hip_Pitch_Right', 'RightLeg': 'Shank_Right', 'RightToeBase': 'right_foot_sphere_5_link', 'LeftArm': 'AL1', 'LeftForeArm': 'left_hand_link', 'LeftHandMiddle3': 'left_hand_sphere_link', 'RightArm': 'AR1', 'RightForeArm': 'right_hand_link', 'RightHandMiddle3': 'right_hand_sphere_link', 'LeftFoot': 'Ankle_Cross_Left', 'RightFoot': 'Ankle_Cross_Right'}, 'lafan1': {'Spine1': 'Trunk', 'LeftUpLeg': 'Hip_Pitch_Left', 'RightUpLeg': 'Hip_Pitch_Right', 'LeftLeg': 'Shank_Left', 'RightLeg': 'Shank_Right', 'LeftArm': 'AL1', 'RightArm': 'AR1', 'LeftForeArm': 'left_hand_link', 'RightForeArm': 'right_hand_link', 'LeftFoot': 'Ankle_Cross_Left', 'RightFoot': 'Ankle_Cross_Right', 'LeftToeBase': 'left_foot_sphere_5_link', 'RightToeBase': 'right_foot_sphere_5_link', 'LeftHand': 'left_hand_sphere_link', 'RightHand': 'right_hand_sphere_link'}}

TOE_NAMES_BY_FORMAT module-attribute

TOE_NAMES_BY_FORMAT = {'lafan1': ['LeftToeBase', 'RightToeBase'], 'smplh': ['L_Toe', 'R_Toe'], 'mocap': ['LeftToeBase', 'RightToeBase'], 'smplx': ['L_Foot', 'R_Foot']}

DATA_FORMAT_CONSTANTS module-attribute

DATA_FORMAT_CONSTANTS: dict[str, dict[str, float]] = {'lafan1': {'default_scale_factor': 1.27 / 1.7}, 'mocap': {'default_human_height': 1.78}}

DexRetargeter

Eden RetargetingBase wrapper around the dex-retargeting port.

Combines one of three SLSQP optimizers (position / vector / dexpilot) with the upstream SeqRetargeting semantics (mean-of-limits warm start, output low-pass filter) into a single DexRetargeter entity-extension.

Typical usage:

.. code-block:: python

retargeter = DexRetargeter(
    env=env,
    options=DexRetargeter.configure(
        entity_name="robot",
        type="vector",
        target_origin_link_names=["wrist", "wrist", "wrist", "wrist"],
        target_task_link_names=["index_tip", "middle_tip", "ring_tip", "pinky_tip"],
        target_link_human_indices=np.array([[0, 0, 0, 0], [8, 12, 16, 20]]),
        scaling_factor=1.6,
    ),
)
qpos = retargeter.retarget(ref_value)              # one frame
qpos_seq = retargeter.retarget_motion(ref_seq)     # batch

Attributes

Classes

DexRetargeter

DexRetargeter(env: 'EnvBase', options: RetargeterOptions)

Bases: RetargetingBase

Port of dex-retargeting backed by Genesis kinematics.

PARAMETER	DESCRIPTION
type	Retargeting algorithm to instantiate. TYPE: ('position', 'vector', 'dexpilot') DEFAULT: "position"
target_joint_names	Subset of entity.dofs_name to optimize. None = all DOFs. TYPE: list[str] or None
target_link_human_indices	Indices into the human hand keypoint array. - position: shape (n,) — one keypoint per target link. - vector : shape (2, n) — origin indices on row 0, task on row 1. - dexpilot: leave None; defaults are derived from the fingertip count. TYPE: ndarray or None
target_link_names	Position-retargeting only. Robot links whose world positions match the human keypoints selected by target_link_human_indices. TYPE: list[str] or None
target_origin_link_names	Vector-retargeting only. Each vector goes from origin to task. TYPE: list[str] or None
target_task_link_names	Vector-retargeting only. Each vector goes from origin to task. TYPE: list[str] or None
finger_tip_link_names	DexPilot only. Five (or fewer) fingertip links. TYPE: list[str] or None
wrist_link_name	DexPilot only. The wrist / palm reference link. TYPE: str or None
scaling_factor	Human-to-robot scale (applies to vector and dexpilot). TYPE: float
huber_delta	Huber loss beta and quadratic regularization weight on (x - last_qpos). TYPE: float
normal_delta	Huber loss beta and quadratic regularization weight on (x - last_qpos). TYPE: float
project_dist	DexPilot fingertip projection thresholds. TYPE: float
escape_dist	DexPilot fingertip projection thresholds. TYPE: float
has_joint_limits	Pass entity DOF limits to nlopt as bound constraints. TYPE: bool
low_pass_alpha	Output smoothing in [0, 1]. 1 = passthrough, 0 = freeze; a value outside [0, 1] disables it. TYPE: float

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

def __init__(self, env: "EnvBase", options: RetargeterOptions) -> None:
    super().__init__(env, options)

    self.optimizer: Optimizer = self._build_optimizer()

    # Joint-limit bounds for the target DOFs. Either pull them from the
    # entity (and forward to nlopt) or use a wide band that effectively
    # disables the constraint.
    if self.has_joint_limits:
        # entity.get_dofs_limit() returns (lower, upper), each (n_envs, n_dofs).
        # Take env 0 and stack to (n_dofs, 2).
        entity_limits = torch.stack(self.entity.get_dofs_limit(), dim=-1)[0].cpu().numpy()
        self.joint_limits = entity_limits[self.optimizer.idx_target]
        self.optimizer.set_joint_limit(self.joint_limits)
    else:
        self.joint_limits = np.tile([-1e4, 1e4], (self.optimizer.opt_dof, 1)).astype(np.float64)

    self.last_qpos = self.joint_limits.mean(axis=1).astype(np.float32)

    self.filter: Optional[LPFilter] = LPFilter(self.low_pass_alpha) if 0 <= self.low_pass_alpha <= 1 else None

Attributes

type class-attribute instance-attribute

type: Literal['position', 'vector', 'dexpilot'] = 'vector'

target_joint_names class-attribute instance-attribute

target_joint_names: Optional[list[str]] = None

target_link_human_indices class-attribute instance-attribute

target_link_human_indices: Optional[ndarray] = None

target_link_names class-attribute instance-attribute

target_link_names: Optional[list[str]] = None

target_origin_link_names class-attribute instance-attribute

target_origin_link_names: Optional[list[str]] = None

target_task_link_names class-attribute instance-attribute

target_task_link_names: Optional[list[str]] = None

finger_tip_link_names class-attribute instance-attribute

finger_tip_link_names: Optional[list[str]] = None

wrist_link_name class-attribute instance-attribute

wrist_link_name: Optional[str] = None

scaling_factor class-attribute instance-attribute

scaling_factor: float = 1.0

huber_delta class-attribute instance-attribute

huber_delta: float = 0.02

normal_delta class-attribute instance-attribute

normal_delta: float = 0.004

project_dist class-attribute instance-attribute

project_dist: float = 0.03

escape_dist class-attribute instance-attribute

escape_dist: float = 0.05

has_joint_limits class-attribute instance-attribute

has_joint_limits: bool = True

low_pass_alpha class-attribute instance-attribute

low_pass_alpha: float = 0.1

optimizer instance-attribute

optimizer: Optimizer = self._build_optimizer()

joint_limits instance-attribute

joint_limits = entity_limits[self.optimizer.idx_target]

last_qpos instance-attribute

last_qpos = self.joint_limits.mean(axis=1).astype(np.float32)

filter instance-attribute

filter: Optional[LPFilter] = LPFilter(self.low_pass_alpha) if 0 <= self.low_pass_alpha <= 1 else None

joint_names property

joint_names: list[str]

Full DOF name list of the underlying entity (dofs_name order).

retargeting_type property

retargeting_type: str

Methods:

retarget

retarget(ref_value: ndarray, fixed_qpos: ndarray | None = None) -> ndarray

Retarget one frame and return a full-DOF qpos (in dofs_name order).

ref_value is the optimizer-specific Cartesian reference: positions (n, 3) for position; vectors (n, 3) for vector and dexpilot. fixed_qpos holds the non-target joints (optimizer.idx_fixed order; zeros when None).

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

def retarget(
    self,
    ref_value: np.ndarray,
    fixed_qpos: np.ndarray | None = None,
) -> np.ndarray:
    """Retarget one frame and return a full-DOF qpos (in ``dofs_name`` order).

    ``ref_value`` is the optimizer-specific Cartesian reference: positions
    ``(n, 3)`` for ``position``; vectors ``(n, 3)`` for ``vector`` and
    ``dexpilot``. ``fixed_qpos`` holds the non-target joints (``optimizer.idx_fixed`` order;
    zeros when ``None``).
    """
    if fixed_qpos is None:
        fixed_qpos = np.zeros(len(self.optimizer.idx_fixed), dtype=np.float32)

    qpos = self.optimizer.retarget(
        ref_value=np.asarray(ref_value, dtype=np.float32),
        fixed_qpos=np.asarray(fixed_qpos, dtype=np.float32),
        last_qpos=np.clip(self.last_qpos, self.joint_limits[:, 0], self.joint_limits[:, 1]),
    )
    self.last_qpos = qpos

    robot_qpos = np.zeros(self.optimizer.num_joints, dtype=np.float32)
    robot_qpos[self.optimizer.idx_fixed] = fixed_qpos
    robot_qpos[self.optimizer.idx_target] = qpos

    if self.filter is not None:
        robot_qpos = self.filter.next(robot_qpos)
    return robot_qpos

retarget_motion

retarget_motion(ref_sequence: ndarray, fixed_qpos_sequence: ndarray | None = None) -> ndarray

Retarget a whole trajectory.

PARAMETER	DESCRIPTION
ref_sequence	(T, ...) — per-frame ref_value arrays. TYPE: ndarray
fixed_qpos_sequence	(T, len(idx_fixed)) if any joints are held fixed; otherwise None. TYPE: ndarray DEFAULT: None

RETURNS	DESCRIPTION
qpos	Full-DOF qpos per frame in entity.dofs_name order. TYPE: np.ndarray, shape ``(T, n_dofs)``

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

def retarget_motion(
    self,
    ref_sequence: np.ndarray,
    fixed_qpos_sequence: np.ndarray | None = None,
) -> np.ndarray:
    """Retarget a whole trajectory.

    Parameters
    ----------
    ref_sequence : np.ndarray
        ``(T, ...)`` — per-frame ``ref_value`` arrays.
    fixed_qpos_sequence : np.ndarray, optional
        ``(T, len(idx_fixed))`` if any joints are held fixed; otherwise
        ``None``.

    Returns
    -------
    qpos : np.ndarray, shape ``(T, n_dofs)``
        Full-DOF qpos per frame in ``entity.dofs_name`` order.
    """
    return np.stack(
        [
            self.retarget(
                ref_sequence[t],
                fixed_qpos=fixed_qpos_sequence[t] if fixed_qpos_sequence is not None else None,
            )
            for t in range(len(ref_sequence))
        ]
    )

reset

reset() -> None

Forget previous-frame state (last_qpos, filter).

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

def reset(self) -> None:
    """Forget previous-frame state (``last_qpos``, filter)."""
    self.last_qpos = self.joint_limits.mean(axis=1).astype(np.float32)
    if self.filter is not None:
        self.filter.reset()

set_qpos

set_qpos(robot_qpos: ndarray) -> None

Manually seed last_qpos (e.g. from the simulator's current state).

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

def set_qpos(self, robot_qpos: np.ndarray) -> None:
    """Manually seed ``last_qpos`` (e.g. from the simulator's current state)."""
    self.last_qpos = np.asarray(robot_qpos, dtype=np.float32)[self.optimizer.idx_target]

get_qpos

get_qpos(fixed_qpos: ndarray | None = None) -> ndarray

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

def get_qpos(self, fixed_qpos: np.ndarray | None = None) -> np.ndarray:
    robot_qpos = np.zeros(self.optimizer.num_joints, dtype=np.float32)
    robot_qpos[self.optimizer.idx_target] = self.last_qpos
    if fixed_qpos is not None:
        robot_qpos[self.optimizer.idx_fixed] = fixed_qpos
    return robot_qpos

extract_ref_value staticmethod

extract_ref_value(joint_pos: ndarray, target_link_human_indices: ndarray, retargeting_type: str) -> ndarray

Build the optimizer's ref_value from a 21-keypoint MANO array.

Same logic as upstream's detect_from_video.py snippet so existing MediaPipe pipelines drop in cleanly.

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

@staticmethod
def extract_ref_value(
    joint_pos: np.ndarray,
    target_link_human_indices: np.ndarray,
    retargeting_type: str,
) -> np.ndarray:
    """Build the optimizer's ``ref_value`` from a 21-keypoint MANO array.

    Same logic as upstream's ``detect_from_video.py`` snippet so existing
    MediaPipe pipelines drop in cleanly.
    """
    if retargeting_type == "POSITION":
        return joint_pos[target_link_human_indices, :]
    origin = target_link_human_indices[0, :]
    task = target_link_human_indices[1, :]
    return joint_pos[task, :] - joint_pos[origin, :]

Functions:

operator2mano

operator2mano(hand_type: HandType) -> ndarray

Return the rotation that maps the operator frame to MANO.

Source code in eden/extensions/retargeting/dex_retargeter/retargeter.py

def operator2mano(hand_type: HandType) -> np.ndarray:
    """Return the rotation that maps the operator frame to MANO."""
    return OPERATOR2MANO[hand_type]

SLSQP-based retargeting optimizers, Genesis-backed.

Ported from dex_retargeting.optimizer (AnyTeleop / DexPilot). The algorithm — Huber loss over keypoint positions or relative vectors, optimized with nlopt's SLSQP and analytic gradients built from positional Jacobians — is unchanged. Only the kinematics layer differs:

• Forward kinematics is performed by calling entity.set_qpos(qpos).
• Jacobians come from entity.get_jacobian(link) and are already in world frame (Genesis returns the spatial Jacobian), so the upstream link_rot @ body_jacobian step is dropped.
• Column ordering follows entity.dofs_name (Eden reorders internally).

Free-joint detection and the MimicJointKinematicAdaptor from upstream are intentionally omitted; we expect hands with no base DOFs (fixed-base or welded onto an arm) with mimic constraints already baked into the URDF.

Classes

Optimizer

Optimizer(entity: 'RigidEntity', target_joint_names: List[str], target_link_human_indices: Optional[ndarray])

Base class. Subclasses build the per-frame objective + gradient.

Operates directly on an Eden :class:~eden.entities.rigid.RigidEntity. All DOF arrays use entity.dofs_name order (the same order set_qpos and get_jacobian's columns use).

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def __init__(
    self,
    entity: "RigidEntity",
    target_joint_names: List[str],
    target_link_human_indices: Optional[np.ndarray],
) -> None:
    # The requirement is that the optimised qpos carries no base DOFs — true for a fixed
    # base and equally for a hand welded onto an arm (``attach`` removes its root joint),
    # so welded dexterous hands are supported here.
    if not entity.has_immobile_base:
        raise ValueError(
            "DexRetargeter optimizers require an entity with no base DOFs (fixed-base, or "
            "welded onto another entity). Floating-hand retargeting (upstream's "
            "`add_dummy_free_joint`) is not yet supported."
        )
    self.entity = entity
    self.device = entity.device
    self.dofs_name: list[str] = list(entity.dofs_name)
    self.num_joints: int = len(self.dofs_name)

    idx_target: list[int] = []
    for name in target_joint_names:
        if name not in self.dofs_name:
            raise ValueError(f"Joint {name!r} not in entity.dofs_name. Available: {self.dofs_name}")
        idx_target.append(self.dofs_name.index(name))
    target_set = set(idx_target)
    self.target_joint_names = target_joint_names
    self.idx_target = np.array(idx_target, dtype=int)
    self.idx_fixed = np.array([i for i in range(self.num_joints) if i not in target_set], dtype=int)

    self.opt = nlopt.opt(nlopt.LD_SLSQP, len(idx_target))
    self.opt_dof = len(idx_target)

    self.target_link_human_indices = target_link_human_indices

Attributes

retargeting_type class-attribute instance-attribute

retargeting_type = 'BASE'

entity instance-attribute

entity = entity

device instance-attribute

device = entity.device

dofs_name instance-attribute

dofs_name: list[str] = list(entity.dofs_name)

num_joints instance-attribute

num_joints: int = len(self.dofs_name)

target_joint_names instance-attribute

target_joint_names = target_joint_names

idx_target instance-attribute

idx_target = np.array(idx_target, dtype=int)

idx_fixed instance-attribute

idx_fixed = np.array([i for i in range(self.num_joints) if i not in target_set], dtype=int)

opt instance-attribute

opt = nlopt.opt(nlopt.LD_SLSQP, len(idx_target))

opt_dof instance-attribute

opt_dof = len(idx_target)

target_link_human_indices instance-attribute

target_link_human_indices = target_link_human_indices

Methods:

set_joint_limit

set_joint_limit(joint_limits: ndarray, epsilon: float = 0.001) -> None

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def set_joint_limit(self, joint_limits: np.ndarray, epsilon: float = 1e-3) -> None:
    if joint_limits.shape != (self.opt_dof, 2):
        raise ValueError(f"Expected joint_limits shape {(self.opt_dof, 2)}, got {joint_limits.shape}")
    self.opt.set_lower_bounds((joint_limits[:, 0] - epsilon).tolist())
    self.opt.set_upper_bounds((joint_limits[:, 1] + epsilon).tolist())

retarget

retarget(ref_value: ndarray, fixed_qpos: ndarray, last_qpos: ndarray) -> ndarray

Run one SLSQP solve and return the optimized target-joint qpos.

PARAMETER	DESCRIPTION
ref_value	Optimizer-specific Cartesian reference (positions or vectors). TYPE: ndarray
fixed_qpos	Values for joints NOT in target_joint_names, ordered to match self.idx_fixed (i.e. entity.dofs_name filtered by that index). TYPE: ndarray
last_qpos	Initial guess and regularization target for this frame. TYPE: ndarray

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def retarget(
    self,
    ref_value: np.ndarray,
    fixed_qpos: np.ndarray,
    last_qpos: np.ndarray,
) -> np.ndarray:
    """Run one SLSQP solve and return the optimized target-joint qpos.

    Parameters
    ----------
    ref_value
        Optimizer-specific Cartesian reference (positions or vectors).
    fixed_qpos
        Values for joints NOT in ``target_joint_names``, ordered to match
        ``self.idx_fixed`` (i.e. ``entity.dofs_name`` filtered by that index).
    last_qpos
        Initial guess and regularization target for this frame.
    """
    if len(fixed_qpos) != len(self.idx_fixed):
        raise ValueError(f"Optimizer expects {len(self.idx_fixed)} fixed joints, got {len(fixed_qpos)}.")
    objective_fn = self.get_objective_function(
        ref_value,
        fixed_qpos,
        np.asarray(last_qpos, dtype=np.float32),
    )
    self.opt.set_min_objective(objective_fn)
    try:
        qpos = self.opt.optimize(last_qpos)
        return np.asarray(qpos, dtype=np.float32)
    except RuntimeError as e:  # nlopt raises this on solver failure
        en.logger.warning(f"nlopt solver failure: {e}")
        return np.asarray(last_qpos, dtype=np.float32)

get_objective_function abstractmethod

get_objective_function(ref_value: ndarray, fixed_qpos: ndarray, last_qpos: ndarray)

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

@abstractmethod
def get_objective_function(
    self,
    ref_value: np.ndarray,
    fixed_qpos: np.ndarray,
    last_qpos: np.ndarray,
): ...

PositionOptimizer

PositionOptimizer(entity: 'RigidEntity', target_joint_names: List[str], target_link_names: List[str], target_link_human_indices: ndarray, huber_delta: float = 0.02, norm_delta: float = 0.004)

Bases: Optimizer

Match absolute world-frame positions of robot links to keypoints.

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def __init__(
    self,
    entity: "RigidEntity",
    target_joint_names: List[str],
    target_link_names: List[str],
    target_link_human_indices: np.ndarray,
    huber_delta: float = 0.02,
    norm_delta: float = 4e-3,
) -> None:
    super().__init__(entity, target_joint_names, target_link_human_indices)
    self.huber_loss = torch.nn.SmoothL1Loss(beta=huber_delta)
    self.norm_delta = norm_delta
    self._target_links = self._cache_links(target_link_names)
    self.opt.set_ftol_abs(1e-5)

Attributes

retargeting_type class-attribute instance-attribute

retargeting_type = 'POSITION'

huber_loss instance-attribute

huber_loss = torch.nn.SmoothL1Loss(beta=huber_delta)

norm_delta instance-attribute

norm_delta = norm_delta

Methods:

get_objective_function

get_objective_function(target_pos, fixed_qpos, last_qpos)

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def get_objective_function(self, target_pos, fixed_qpos, last_qpos):
    expected = (len(self._target_links), 3)
    if target_pos.shape != expected:
        raise ValueError(
            f"PositionOptimizer: target_pos shape {target_pos.shape} does not match "
            f"{expected} (one (x, y, z) per target link). Did you pass the full "
            f"21-keypoint MANO array? Slice by target_link_human_indices first, "
            f"e.g. via DexRetargeter.extract_ref_value."
        )
    qpos = np.zeros(self.num_joints, dtype=np.float32)
    qpos[self.idx_fixed] = fixed_qpos
    torch_target_pos = torch.as_tensor(target_pos, dtype=torch.float32, device=self.device)
    torch_target_pos.requires_grad_(False)

    def objective(x: np.ndarray, grad: np.ndarray) -> float:
        qpos[self.idx_target] = x
        self._set_qpos(qpos)
        body_pos = self._link_positions(self._target_links)

        torch_body_pos = torch.as_tensor(body_pos, dtype=torch.float32, device=self.device)
        torch_body_pos.requires_grad_()
        huber_distance = self.huber_loss(torch_body_pos, torch_target_pos)
        result = float(huber_distance.detach().item())

        if grad.size > 0:
            self._write_grad(self._target_links, torch_body_pos, huber_distance, x, last_qpos, grad)
        return result

    return objective

VectorOptimizer

VectorOptimizer(entity: 'RigidEntity', target_joint_names: List[str], target_origin_link_names: List[str], target_task_link_names: List[str], target_link_human_indices: ndarray, huber_delta: float = 0.02, norm_delta: float = 0.004, scaling: float = 1.0)

Bases: Optimizer

Match relative vectors between robot link pairs to keypoint pairs.

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def __init__(
    self,
    entity: "RigidEntity",
    target_joint_names: List[str],
    target_origin_link_names: List[str],
    target_task_link_names: List[str],
    target_link_human_indices: np.ndarray,
    huber_delta: float = 0.02,
    norm_delta: float = 4e-3,
    scaling: float = 1.0,
) -> None:
    super().__init__(entity, target_joint_names, target_link_human_indices)
    self.huber_loss = torch.nn.SmoothL1Loss(beta=huber_delta, reduction="mean")
    self.norm_delta = norm_delta
    self.scaling = scaling
    self._setup_link_pairs(target_origin_link_names, target_task_link_names)
    self.opt.set_ftol_abs(1e-6)

Attributes

retargeting_type class-attribute instance-attribute

retargeting_type = 'VECTOR'

huber_loss instance-attribute

huber_loss = torch.nn.SmoothL1Loss(beta=huber_delta, reduction='mean')

norm_delta instance-attribute

norm_delta = norm_delta

scaling instance-attribute

scaling = scaling

Methods:

get_objective_function

get_objective_function(target_vector, fixed_qpos, last_qpos)

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def get_objective_function(self, target_vector, fixed_qpos, last_qpos):
    expected = (len(self.origin_link_indices), 3)
    if target_vector.shape != expected:
        raise ValueError(
            f"VectorOptimizer: target_vector shape {target_vector.shape} does not "
            f"match {expected} (one (dx, dy, dz) per origin/task link pair). Did "
            f"you pass the full 21-keypoint MANO array? Compute "
            f"`kp[task_idx] - kp[origin_idx]` first, e.g. via "
            f"DexRetargeter.extract_ref_value."
        )
    qpos = np.zeros(self.num_joints, dtype=np.float32)
    qpos[self.idx_fixed] = fixed_qpos
    torch_target_vec = torch.as_tensor(target_vector, dtype=torch.float32, device=self.device) * self.scaling
    torch_target_vec.requires_grad_(False)

    def objective(x: np.ndarray, grad: np.ndarray) -> float:
        qpos[self.idx_target] = x
        self._set_qpos(qpos)
        body_pos = self._link_positions(self._computed_links)

        torch_body_pos = torch.as_tensor(body_pos, dtype=torch.float32, device=self.device)
        torch_body_pos.requires_grad_()
        robot_vec = torch_body_pos[self.task_link_indices, :] - torch_body_pos[self.origin_link_indices, :]
        vec_dist = torch.norm(robot_vec - torch_target_vec, dim=1)
        huber_distance = self.huber_loss(vec_dist, torch.zeros_like(vec_dist))
        result = float(huber_distance.detach().item())

        if grad.size > 0:
            self._write_grad(self._computed_links, torch_body_pos, huber_distance, x, last_qpos, grad)
        return result

    return objective

DexPilotOptimizer

DexPilotOptimizer(entity: 'RigidEntity', target_joint_names: List[str], finger_tip_link_names: List[str], wrist_link_name: str, target_link_human_indices: Optional[ndarray] = None, huber_delta: float = 0.03, norm_delta: float = 0.004, project_dist: float = 0.03, escape_dist: float = 0.05, eta1: float = 0.0001, eta2: float = 0.03, scaling: float = 1.0)

Bases: Optimizer

DexPilot vector retargeting with thumb-to-fingertip projection.

A direct port of dex_retargeting.optimizer.DexPilotOptimizer. Generates its own link pairs (each fingertip-to-wrist + every fingertip-pair) and snaps thumb-to-fingertip distances inside project_dist to encourage pinch grasps. See https://arxiv.org/abs/1910.03135.

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def __init__(
    self,
    entity: "RigidEntity",
    target_joint_names: List[str],
    finger_tip_link_names: List[str],
    wrist_link_name: str,
    target_link_human_indices: Optional[np.ndarray] = None,
    huber_delta: float = 0.03,
    norm_delta: float = 4e-3,
    project_dist: float = 0.03,
    escape_dist: float = 0.05,
    eta1: float = 1e-4,
    eta2: float = 3e-2,
    scaling: float = 1.0,
) -> None:
    if not (2 <= len(finger_tip_link_names) <= 5):
        raise ValueError(f"DexPilot supports 2..5 fingertips; got {len(finger_tip_link_names)}.")
    self.num_fingers = len(finger_tip_link_names)

    origin_link_index, task_link_index = self._generate_link_indices(self.num_fingers)
    if target_link_human_indices is None:
        target_link_human_indices = (np.stack([origin_link_index, task_link_index], axis=0) * 4).astype(int)

    links_name = [wrist_link_name] + list(finger_tip_link_names)
    origin_names = [links_name[i] for i in origin_link_index]
    task_names = [links_name[i] for i in task_link_index]

    super().__init__(entity, target_joint_names, target_link_human_indices)
    self.scaling = scaling
    self.huber_loss = torch.nn.SmoothL1Loss(beta=huber_delta, reduction="none")
    self.norm_delta = norm_delta
    self.project_dist = project_dist
    self.escape_dist = escape_dist
    self.eta1 = eta1
    self.eta2 = eta2

    self._setup_link_pairs(origin_names, task_names)
    self.opt.set_ftol_abs(1e-6)

    (
        self.projected,
        self.s2_project_index_origin,
        self.s2_project_index_task,
        self.projected_dist,
    ) = self._set_dexpilot_cache(self.num_fingers, eta1, eta2)

Attributes

retargeting_type class-attribute instance-attribute

retargeting_type = 'DEXPILOT'

num_fingers instance-attribute

num_fingers = len(finger_tip_link_names)

scaling instance-attribute

scaling = scaling

huber_loss instance-attribute

huber_loss = torch.nn.SmoothL1Loss(beta=huber_delta, reduction='none')

norm_delta instance-attribute

norm_delta = norm_delta

project_dist instance-attribute

project_dist = project_dist

escape_dist instance-attribute

escape_dist = escape_dist

eta1 instance-attribute

eta1 = eta1

eta2 instance-attribute

eta2 = eta2

Methods:

get_objective_function

get_objective_function(target_vector, fixed_qpos, last_qpos)

Source code in eden/extensions/retargeting/dex_retargeter/optimizer.py

def get_objective_function(self, target_vector, fixed_qpos, last_qpos):
    qpos = np.zeros(self.num_joints, dtype=np.float32)
    qpos[self.idx_fixed] = fixed_qpos

    len_proj = len(self.projected)
    len_s2 = len(self.s2_project_index_task)
    len_s1 = len_proj - len_s2

    target_vec_dist = np.linalg.norm(target_vector[:len_proj], axis=1)
    self.projected[:len_s1][target_vec_dist[0:len_s1] < self.project_dist] = True
    self.projected[:len_s1][target_vec_dist[0:len_s1] > self.escape_dist] = False
    self.projected[len_s1:len_proj] = np.logical_and(
        self.projected[:len_s1][self.s2_project_index_origin],
        self.projected[:len_s1][self.s2_project_index_task],
    )
    self.projected[len_s1:len_proj] = np.logical_and(
        self.projected[len_s1:len_proj], target_vec_dist[len_s1:len_proj] <= _DEXPILOT_SECONDARY_PROJECT_DIST
    )

    normal_weight = np.ones(len_proj, dtype=np.float32)
    high_weight = np.array([200] * len_s1 + [400] * len_s2, dtype=np.float32)
    weight = np.where(self.projected, high_weight, normal_weight)
    # `as_tensor(..., device=self.device)`, not `from_numpy`: every tensor in this objective is built on
    # the entity's device, explicitly, since there is no torch default device (Genesis stopped setting one in
    # https://github.com/Genesis-Embodied-AI/Genesis/pull/3431). `from_numpy` aliases the numpy buffer, so it is
    # always CPU: on a GPU backend, multiplying the CUDA huber term by it raises inside nlopt, and `retarget`'s
    # `except RuntimeError` turns that into a per-frame warning while handing back `last_qpos` -- a robot hand
    # that silently never moves.
    weight = torch.as_tensor(
        np.concatenate([weight, np.ones(self.num_fingers, dtype=np.float32) * len_proj + self.num_fingers]),
        dtype=torch.float32,
        device=self.device,
    )

    normal_vec = target_vector * self.scaling
    dir_vec = target_vector[:len_proj] / (target_vec_dist[:, None] + 1e-6)
    projected_vec = dir_vec * self.projected_dist[:, None]
    reference_vec = np.where(self.projected[:, None], projected_vec, normal_vec[:len_proj])
    reference_vec = np.concatenate([reference_vec, normal_vec[len_proj:]], axis=0)
    torch_target_vec = torch.as_tensor(reference_vec, dtype=torch.float32, device=self.device)
    torch_target_vec.requires_grad_(False)

    def objective(x: np.ndarray, grad: np.ndarray) -> float:
        qpos[self.idx_target] = x
        self._set_qpos(qpos)
        body_pos = self._link_positions(self._computed_links)

        torch_body_pos = torch.as_tensor(body_pos, dtype=torch.float32, device=self.device)
        torch_body_pos.requires_grad_()
        robot_vec = torch_body_pos[self.task_link_indices, :] - torch_body_pos[self.origin_link_indices, :]
        vec_dist = torch.norm(robot_vec - torch_target_vec, dim=1)
        huber_distance = (self.huber_loss(vec_dist, torch.zeros_like(vec_dist)) * weight / robot_vec.shape[0]).sum()
        result = float(huber_distance.detach().item())

        if grad.size > 0:
            self._write_grad(self._computed_links, torch_body_pos, huber_distance, x, last_qpos, grad)
        return result

    return objective

Bundled robot + retargeter presets for dex-retargeting.

Single source of truth shared by the eden retarget dex CLI and the examples/recipes/retargeting/dex_retargeter scripts. Each :class:HandPreset pairs a RobotOptions factory with a RetargeterOptions factory. The RetargeterOptions factories import DexRetargeter lazily (it pulls nlopt), so importing this module for :data:POSITION_PRESET_NAMES — used to build the CLI's --preset choices — stays free of the dex-retargeter extra.

Fingertip link names and MANO keypoint indices are read off the robot's MetadataOptions (target_hand_links / mano_hand_idx), so adding a hand only needs a new entry in :data:HAND_PRESETS.

Attributes

HAND_PRESETS module-attribute

HAND_PRESETS: dict[str, HandPreset] = {'xhand1_right_dexpilot': HandPreset(_xhand1_r_robot, _xhand1_right_dexpilot, 'right'), 'xhand1_right_vector': HandPreset(_xhand1_r_robot, _xhand1_right_vector, 'right'), 'xhand1_right_position': HandPreset(_xhand1_r_robot, _xhand1_right_position, 'right'), 'wuji_hand2beta2_right_dexpilot': HandPreset(partial(_wuji_hand2_robot, 'right'), partial(_wuji_hand2_dexpilot, 'right'), 'right'), 'wuji_hand2beta2_left_dexpilot': HandPreset(partial(_wuji_hand2_robot, 'left'), partial(_wuji_hand2_dexpilot, 'left'), 'left')}

POSITION_PRESET_NAMES module-attribute

POSITION_PRESET_NAMES: tuple[str, ...] = ('xhand1_right_position',)

Classes

HandPreset dataclass

HandPreset(robot_options_factory: Callable[[], 'RobotOptions'], retargeter_options: Callable[[], 'RetargeterOptions'], side: Literal['left', 'right'])

A robot-options factory paired with its retargeter-options factory, for one side of hand.

Attributes

robot_options_factory instance-attribute

robot_options_factory: Callable[[], 'RobotOptions']

retargeter_options instance-attribute

retargeter_options: Callable[[], 'RetargeterOptions']

side instance-attribute

side: Literal['left', 'right']

Methods:

URDF↔MANO frame alignment, derived from Eden's up / front fields.

A hand URDF's up and front declarations encode the same kind of "asset-frame → world-reference" rotation that :func:eden.utils.geom.align_up_and_front already produces for any Eden entity. For hand robots the convention is:

• up = the URDF axis along which the fingers extend
• front = the URDF axis along which the palm normal points (= the URDF axis that aligns with world +X after load)

Eden's defaults (up=(0,0,1), front=(1,0,0)) already match XHand1_R's URDF (fingers along URDF +Z, palm normal along URDF +X), so XHand1_R loads identity. MANO's right-hand canonical frame is the matching declaration in MANO-local axes:

• fingers (= up) along -X in MANO local
• palm normal (= front) along -Y in MANO local

Applying align_up_and_front twice (once for the robot, once for MANO) gives a URDF → canonical-world → MANO chain that the retargeter can compose into a single rotation — no per-robot urdf_to_mano_quat metadata needed.

This module exposes :func:urdf_to_mano_rot, which returns:

• R_mano_to_world — used to rotate MANO-local keypoints into the retargeting (= canonical world) frame so the optimizer's targets line up with the robot's actually-loaded fingertip workspace.
• q_urdf_to_mano — composed onto the per-frame MANO wrist quat to get the floating-base base_quat_w that places the robot at the right world orientation for the saved dofs_pos to reproduce MANO fingertips at viz time.

Classes

Functions:

urdf_to_mano_rot

urdf_to_mano_rot(robot: 'RigidEntity') -> tuple[ndarray, ndarray]

Derive the URDF↔MANO conversion from the robot's up / front.

Returns (R_mano_to_world, q_urdf_to_mano) where:

• R_mano_to_world (3x3) — maps MANO-local vectors into the canonical world frame the robot is loaded into via align_up_and_front(robot.up, robot.front). Used at retargeting time to bring MANO keypoint targets into the robot's world frame.
• q_urdf_to_mano (4,) wxyz — composed onto the per-frame MANO wrist quat to produce the floating-base base_quat_w for viz.

The robot's MANO side is read from metadata.side (defaults to right).

Source code in eden/extensions/retargeting/dex_retargeter/calibrate.py

def urdf_to_mano_rot(robot: "RigidEntity") -> tuple[np.ndarray, np.ndarray]:
    """Derive the URDF↔MANO conversion from the robot's ``up`` / ``front``.

    Returns ``(R_mano_to_world, q_urdf_to_mano)`` where:

    - ``R_mano_to_world`` (3x3) — maps MANO-local vectors into the
      canonical world frame the robot is loaded into via
      ``align_up_and_front(robot.up, robot.front)``. Used at retargeting
      time to bring MANO keypoint targets into the robot's world frame.
    - ``q_urdf_to_mano`` (4,) wxyz — composed onto the per-frame MANO
      wrist quat to produce the floating-base ``base_quat_w`` for viz.

    The robot's MANO side is read from ``metadata.side`` (defaults to right).
    """
    from genesis.utils.geom import R_to_quat

    md = getattr(robot.options, "metadata", None)
    side = getattr(md, "side", "right") if md is not None else "right"
    mano_up, mano_front = _MANO_UP_FRONT[side]

    R_robot_to_world = align_up_and_front(robot.options.up, robot.options.front)
    R_mano_to_world = align_up_and_front(mano_up, mano_front)
    # R_urdf_to_mano = R_world_to_mano @ R_urdf_to_world = R_mano_to_world.T @ R_robot_to_world
    R_urdf_to_mano = (R_mano_to_world.T @ R_robot_to_world).astype(np.float32)
    q_urdf_to_mano = R_to_quat(R_urdf_to_mano).astype(np.float32)
    return R_mano_to_world.astype(np.float32), q_urdf_to_mano

Constants for the dex-retargeting port.

Mirrors dex_retargeting.constants but trimmed to what the Eden port actually consumes. Robot-name enums upstream are tied to YAML config files shipped with the package; in Eden we plug in any RigidEntity directly, so those enums are not needed.

Attributes

OPERATOR2MANO_RIGHT module-attribute

OPERATOR2MANO_RIGHT = np.array([[0, 0, -1], [-1, 0, 0], [0, 1, 0]])

OPERATOR2MANO_LEFT module-attribute

OPERATOR2MANO_LEFT = np.array([[0, 0, -1], [1, 0, 0], [0, -1, 0]])

OPERATOR2MANO module-attribute

OPERATOR2MANO = {HandType.right: OPERATOR2MANO_RIGHT, HandType.left: OPERATOR2MANO_LEFT}

Classes

HandType

Bases: Enum

Attributes

right class-attribute instance-attribute

right = enum.auto()

left class-attribute instance-attribute

left = enum.auto()

Helpers for the dex-retargeting port (e.g. low-pass filter).

Classes

LPFilter

LPFilter(alpha: float)

Exponential-moving-average low-pass filter.

alpha close to 1 means almost-passthrough; small alpha produces heavier smoothing (and higher latency).

Source code in eden/extensions/retargeting/dex_retargeter/utils.py

def __init__(self, alpha: float) -> None:
    self.alpha = float(alpha)
    self.y: np.ndarray | None = None
    self.is_init: bool = False

Attributes

alpha instance-attribute

alpha = float(alpha)

y instance-attribute

y: ndarray | None = None

is_init instance-attribute

is_init: bool = False

Methods:

next

next(x: ndarray) -> ndarray

Source code in eden/extensions/retargeting/dex_retargeter/utils.py

def next(self, x: np.ndarray) -> np.ndarray:
    if not self.is_init:
        self.y = np.array(x, copy=True)
        self.is_init = True
        return self.y.copy()
    self.y = self.y + self.alpha * (x - self.y)
    return self.y.copy()

reset

reset() -> None

Source code in eden/extensions/retargeting/dex_retargeter/utils.py

def reset(self) -> None:
    self.y = None
    self.is_init = False

OmniRetarget

OmniRetarget optimization-based retargeter.

Classes

OmniRetargeter

OmniRetargeter(env, options: RetargeterOptions)

Bases: RetargetingBase

Interaction-mesh retargeter backed by Eden/Genesis.

Works on a floating-base entity (qpos [pos, quat_wxyz, joints]) and on a fixed-base or welded one (qpos joints only). Every name below is checked at construction: a link, DOF, demo joint or foot_links key the entity or config does not have raises, naming it.

PARAMETER	DESCRIPTION
foot_links	{virtual_name: entity_link_name} for foot contact points. A plain list [link_name, ...] is also accepted (virtual name = link name). TYPE: dict[str, str]
demo_joints	Joint names of the demo motion's joint axis (e.g. SMPL-H). TYPE: list[str]
laplacian_match_links	{demo_joint_name: entity_link_name} for the Laplacian mesh. TYPE: dict[str, str]
link_offsets	{name: local_offset_xyz} for virtual offset points. TYPE: dict[str, ndarray]
manual_lb	Manual joint-limit overrides keyed by DOF name. TYPE: dict[str, float]
manual_ub	Manual joint-limit overrides keyed by DOF name. TYPE: dict[str, float]
manual_cost	Per-joint cost weights keyed by DOF name. TYPE: dict[str, float]
nominal_tracking_dofs	DOF names for nominal-pose tracking. TYPE: list[str]
foot_sticking_map	Maps sticking flag keys (e.g. "L_Toe") to lists of foot_links keys to constrain when that flag is active. TYPE: dict[str, list[str]]
laplacian_weights	Weight of every interaction-mesh vertex's Laplacian-deformation term. TYPE: float
smooth_weight	Weight pulling each frame's solution toward the previous frame's. TYPE: float
step_size	Trust-region radius for SQP. TYPE: float
foot_sticking_tolerance	TYPE: float
foot_z_sticking_tolerance	Tolerance for optional Z sticking during active foot contact. TYPE: float
foot_ground_z	World-frame ground plane height used by optional foot-floor constraint. TYPE: float
foot_ground_margin	Extra non-penetration margin above foot_ground_z. TYPE: float
collision_detection_threshold	TYPE: float
penetration_tolerance	TYPE: float
w_nominal_tracking_init	TYPE: float
nominal_tracking_tau	Exponential decay constant for nominal tracking weight. TYPE: float
activate_foot_sticking	TYPE: bool
activate_foot_z_sticking	Keep a sticking foot's z close to the previous frame's, to reduce vertical jitter during stance. TYPE: bool
activate_foot_ground_constraint	Bound every foot_links point above foot_ground_z + foot_ground_margin on every frame. TYPE: bool
activate_obj_non_penetration	TYPE: bool
activate_joint_limits	TYPE: bool
w_root	Weight of the root-position tracking term (floating base only; the first laplacian_match_links entry is the root). TYPE: float
n_iter_init	Maximum SQP iterations on the first frame and on every later one (both >= 1). TYPE: int
n_iter	Maximum SQP iterations on the first frame and on every later one (both >= 1). TYPE: int

Source code in eden/extensions/retargeting/omniretarget/retargeter.py

def __init__(self, env, options: RetargeterOptions) -> None:
    super().__init__(env, options)

    # -- normalize mutable fields ------------------------------------------
    if isinstance(self.foot_links, list):
        self.foot_links = {k: k for k in self.foot_links}
    for name in ("n_iter_init", "n_iter"):
        if getattr(self, name) < 1:
            raise ValueError(
                f"OmniRetargeter: {name}={getattr(self, name)!r} runs no SQP iteration, so no frame would "
                f"be solved. Pass {name} >= 1 (the defaults are n_iter_init=50, n_iter=10)."
            )
    self._refuse_unknown(
        "laplacian_match_links keys",
        self.laplacian_match_links,
        self.demo_joints,
        "demo_joints (the demo's joint axis)",
    )
    self.smplh_mapped_joint_indices: list[int] = [
        self.demo_joints.index(name) for name in self.laplacian_match_links
    ]

    # -- validate link names against entity ---------------------------------
    available_links = [link.name for link in self.entity.links]
    self._refuse_unknown(
        "laplacian_match_links values",
        self.laplacian_match_links.values(),
        available_links,
        f"the links of entity '{self.entity_name}'",
    )
    self._refuse_unknown(
        "foot_links values", self.foot_links.values(), available_links, f"the links of entity '{self.entity_name}'"
    )
    self._refuse_unknown(
        "foot_sticking_map entries",
        [key for keys in self.foot_sticking_map.values() for key in keys],
        self.foot_links,
        "the foot_links keys",
    )

    # -- DOF layout --------------------------------------------------------
    #
    # Eden floating-base DOF layout:
    #   qpos:  [base_pos(3), base_quat(4), joints(N)]  → total 7 + N
    #   DOF:   [base_lin(3), base_ang(3),  joints(N)]  → total 6 + N
    #
    # The optimisation variable ``dqa`` lives in DOF (tangent) space.
    # ---------------------------------------------------------------
    self.dofs_name: list[str] = list(self.entity.dofs_name)
    self.n_joints: int = len(self.entity.dofs_name)
    self.nq_opt: int = self.entity.n_dofs
    # 6 base tangent DOFs only when a free root exists. ``has_immobile_base`` (not
    # ``is_fixed_base``) is the question: a welded child's root joint is removed by
    # ``attach``, so its qpos and DOF vectors have no base prefix either.
    self._joint_dof_start: int = 0 if self.entity.has_immobile_base else 6
    # The same split in qpos: ``[pos(3), quat_wxyz(4), joints]``, or just ``joints``.
    self._joint_q_start: int = 0 if self.entity.has_immobile_base else 7
    self.nq: int = self._joint_q_start + self.n_joints
    # ``_integrate_dqa`` adds a step to each joint q, which only holds when every joint has
    # one q and one DOF; a ball joint (4 q / 3 DOFs) or a ``dofs_name`` that leaves joints
    # out would put every slice past it in the wrong place.
    expected_n_dofs = self._joint_dof_start + self.n_joints
    if self.nq != self.entity.n_qs or expected_n_dofs != self.nq_opt:
        multi_dof = sorted({name for name in self.dofs_name if self.dofs_name.count(name) > 1})
        raise ValueError(
            f"OmniRetargeter: entity '{self.entity_name}' has n_qs={self.entity.n_qs} and "
            f"n_dofs={self.nq_opt}, but OmniRetarget handles only "
            f"{'a fixed base' if self.entity.has_immobile_base else 'a free root (7 q / 6 DOFs)'} plus "
            f"one-DOF hinge or slide joints, all listed in dofs_name ({self.n_joints} entries here), "
            f"which would give n_qs={self.nq} and n_dofs={expected_n_dofs}. "
            f"Joints with more than one DOF: {multi_dof or 'none (so dofs_name leaves joints out)'}. "
            f"Retarget an entity built from one-DOF joints, e.g. a ball joint split into three hinges."
        )

    for field in ("manual_lb", "manual_ub", "manual_cost", "nominal_tracking_dofs"):
        self._refuse_unknown(
            field, getattr(self, field), self.dofs_name, f"the dofs_name of entity '{self.entity_name}'"
        )

    # -- joint limits & cost weights (DOF space) ---------------------------
    self._init_limits_and_costs()

    # -- nominal tracking indices (joint DOFs only) ------------------------
    self.nominal_tracking_joint_indices = np.array(
        [self.dofs_name.index(name) for name in self.nominal_tracking_dofs], dtype=int
    )
    self.nominal_tracking_dof_indices = self._joint_dof_start + self.nominal_tracking_joint_indices

Attributes

foot_links class-attribute instance-attribute

foot_links: dict[str, str] = {}

demo_joints class-attribute instance-attribute

demo_joints: list = []

laplacian_match_links class-attribute instance-attribute

laplacian_match_links: dict = {}

link_offsets class-attribute instance-attribute

link_offsets: dict[str, ndarray] = {}

manual_lb class-attribute instance-attribute

manual_lb: dict[str, float] = {}

manual_ub class-attribute instance-attribute

manual_ub: dict[str, float] = {}

manual_cost class-attribute instance-attribute

manual_cost: dict[str, float] = {}

nominal_tracking_dofs class-attribute instance-attribute

nominal_tracking_dofs: list = []

foot_sticking_map class-attribute instance-attribute

foot_sticking_map: dict[str, list[str]] = {}

laplacian_weights class-attribute instance-attribute

laplacian_weights: float = 10.0

smooth_weight class-attribute instance-attribute

smooth_weight: float = 0.2

step_size class-attribute instance-attribute

step_size: float = 0.2

foot_sticking_tolerance class-attribute instance-attribute

foot_sticking_tolerance: float = 0.001

foot_z_sticking_tolerance class-attribute instance-attribute

foot_z_sticking_tolerance: float = 0.005

foot_ground_z class-attribute instance-attribute

foot_ground_z: float = 0.0

foot_ground_margin class-attribute instance-attribute

foot_ground_margin: float = 0.0

collision_detection_threshold class-attribute instance-attribute

collision_detection_threshold: float = 0.1

penetration_tolerance class-attribute instance-attribute

penetration_tolerance: float = 0.001

w_nominal_tracking_init class-attribute instance-attribute

w_nominal_tracking_init: float = 5.0

nominal_tracking_tau class-attribute instance-attribute

nominal_tracking_tau: float = 10.0

activate_foot_sticking class-attribute instance-attribute

activate_foot_sticking: bool = True

activate_foot_z_sticking class-attribute instance-attribute

activate_foot_z_sticking: bool = False

activate_foot_ground_constraint class-attribute instance-attribute

activate_foot_ground_constraint: bool = False

activate_obj_non_penetration class-attribute instance-attribute

activate_obj_non_penetration: bool = True

activate_joint_limits class-attribute instance-attribute

activate_joint_limits: bool = True

w_root class-attribute instance-attribute

w_root: float = 100.0

n_iter_init class-attribute instance-attribute

n_iter_init: int = 50

n_iter class-attribute instance-attribute

n_iter: int = 10

smplh_mapped_joint_indices instance-attribute

smplh_mapped_joint_indices: list[int] = [self.demo_joints.index(name) for name in self.laplacian_match_links]

dofs_name instance-attribute

dofs_name: list[str] = list(self.entity.dofs_name)

n_joints instance-attribute

n_joints: int = len(self.entity.dofs_name)

nq_opt instance-attribute

nq_opt: int = self.entity.n_dofs

nq instance-attribute

nq: int = self._joint_q_start + self.n_joints

nominal_tracking_joint_indices instance-attribute

nominal_tracking_joint_indices = np.array([self.dofs_name.index(name) for name in self.nominal_tracking_dofs], dtype=int)

nominal_tracking_dof_indices instance-attribute

nominal_tracking_dof_indices = self._joint_dof_start + self.nominal_tracking_joint_indices

Methods:

retarget_motion

retarget_motion(human_joint_motions: ndarray, foot_sticking_sequences: list[dict[str, bool]], object_name: str | None = None, object_poses: ndarray | None = None, scale_factor: float = 1.0, q_a_init: ndarray | None = None, q_nominal_list: ndarray | None = None, n_surface_points: int = 100, constant_nominal_weight: bool = True) -> ndarray

Run the full retargeting loop over all frames.

PARAMETER	DESCRIPTION
human_joint_motions	Human joint positions, shape (T, n_demo_joints, 3). TYPE: ndarray
foot_sticking_sequences	{toe_name: bool} per frame. TYPE: list of dicts
object_name	Entity name in the environment (e.g. "largebox"), or None for retargeting without an interaction object. TYPE: str or None DEFAULT: None
object_poses	[pos(3), quat_wxyz(4)]. Required when object_name is not None. TYPE: (T, 7) or None DEFAULT: None
scale_factor	Ratio robot_height / human_height used to scale object surface points to the demo (human) frame. TYPE: float DEFAULT: 1.0
q_a_init	Optional initial qpos, shape (nq,): 7 + n_joints on a floating base, n_joints on a fixed one. Defaults to the first nominal frame when q_nominal_list is given, else to :meth:_compute_q_init. TYPE: ndarray or None DEFAULT: None
q_nominal_list	Optional nominal trajectory, shape (T, nq). TYPE: ndarray or None DEFAULT: None
n_surface_points	Number of surface points to sample from the object entity. TYPE: int DEFAULT: 100
constant_nominal_weight	If True, use a constant nominal tracking weight; if False, apply exponential decay over frames. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
retargeted	TYPE: (T, nq)

Source code in eden/extensions/retargeting/omniretarget/retargeter.py

def retarget_motion(
    self,
    human_joint_motions: np.ndarray,
    foot_sticking_sequences: list[dict[str, bool]],
    object_name: str | None = None,
    object_poses: np.ndarray | None = None,
    scale_factor: float = 1.0,
    q_a_init: np.ndarray | None = None,
    q_nominal_list: np.ndarray | None = None,
    n_surface_points: int = 100,
    constant_nominal_weight: bool = True,
) -> np.ndarray:
    """Run the full retargeting loop over all frames.

    Parameters
    ----------
    human_joint_motions : np.ndarray
        Human joint positions, shape ``(T, n_demo_joints, 3)``.
    foot_sticking_sequences : list of dicts
        ``{toe_name: bool}`` per frame.
    object_name : str or None
        Entity name in the environment (e.g. ``"largebox"``), or
        ``None`` for retargeting without an interaction object.
    object_poses : (T, 7) or None
        ``[pos(3), quat_wxyz(4)]``.  Required when *object_name* is
        not ``None``.
    scale_factor : float
        Ratio ``robot_height / human_height`` used to scale object
        surface points to the demo (human) frame.
    q_a_init : np.ndarray or None
        Optional initial qpos, shape ``(nq,)``: ``7 + n_joints`` on a floating base,
        ``n_joints`` on a fixed one. Defaults to the first nominal frame when
        ``q_nominal_list`` is given, else to :meth:`_compute_q_init`.
    q_nominal_list : np.ndarray or None
        Optional nominal trajectory, shape ``(T, nq)``.
    n_surface_points : int
        Number of surface points to sample from the object entity.
    constant_nominal_weight : bool
        If True, use a constant nominal tracking weight; if False,
        apply exponential decay over frames.

    Returns
    -------
    retargeted : (T, nq)
    """
    num_frames = human_joint_motions.shape[0]
    self._check_motion_inputs(
        human_joint_motions, foot_sticking_sequences, object_name, object_poses, q_a_init, q_nominal_list
    )

    # -- Obtain object surface points from entity --------------------------
    if object_name is not None:
        obj_entity = self.env.entities[object_name]
        points_list = get_entity_surface_points(obj_entity, n_points=n_surface_points)
        object_points_local = np.concatenate([p.cpu().numpy() for p in points_list if p is not None], axis=0)
        object_points_local_demo = object_points_local * scale_factor
    else:
        object_points_local = np.zeros((0, 3))
        object_points_local_demo = np.zeros((0, 3))

    if q_a_init is None:
        if q_nominal_list is not None:
            q_a_init = q_nominal_list[0]
        else:
            q_a_init = self._compute_q_init(human_joint_motions, object_name, object_poses)
    # The running solution: each frame starts from the previous frame's result, which is also
    # what its smoothness and foot-sticking terms are measured against.
    q = np.array(q_a_init, dtype=float)
    retargeted_motions = []

    en.logger.info(f"Starting motion retargeting for {num_frames} frames...")

    for i in tqdm(range(num_frames)):
        if object_poses is not None:
            obj_trans = object_poses[i, :3]
            obj_quat = object_poses[i, 3:]
        else:
            obj_trans = np.zeros(3)
            obj_quat = np.array([1.0, 0.0, 0.0, 0.0])

        human_mapped = human_joint_motions[i, self.smplh_mapped_joint_indices]

        # Transform to object frame when applicable
        if object_name is None:
            human_in_obj = human_mapped
        else:
            human_in_obj = inv_transform_by_trans_quat(human_mapped, obj_trans, obj_quat)

        # Interaction mesh
        all_pts = np.vstack([human_in_obj, object_points_local_demo])
        simplices = compute_delaunay_simplices(all_pts)
        adj_list = get_adjacency_list(simplices, len(all_pts))
        target_lap = calculate_laplacian_matrix(all_pts, adj_list) @ all_pts

        # Nominal weight schedule
        if constant_nominal_weight:
            w_nom = self.w_nominal_tracking_init
        else:
            w_nom = self.w_nominal_tracking_init * np.exp(-i / self.nominal_tracking_tau)

        q_nominal_joints = None
        if q_nominal_list is not None:
            q_nominal_joints = np.asarray(q_nominal_list)[i, self._joint_q_start :]

        q, _cost = self.iterate(
            q_n=q,
            q_t_last=q,
            target_laplacian=target_lap,
            adj_list=adj_list,
            obj_pts_local=object_points_local,
            foot_sticking=foot_sticking_sequences[i],
            w_nominal_tracking=w_nom,
            q_nominal_joints=q_nominal_joints,
            init_t=(i == 0),
            n_iter=self.n_iter_init if i == 0 else self.n_iter,
            obj_pos=obj_trans,
            obj_quat=obj_quat,
            target_root_pos=(human_in_obj[0] if len(human_in_obj) > 0 else None),
            object_name=object_name,
        )

        retargeted_motions.append(q)

    return np.array(retargeted_motions)

iterate

iterate(q_n: ndarray, q_t_last: ndarray, target_laplacian: ndarray, adj_list: list[set], obj_pts_local: ndarray, foot_sticking: dict[str, bool], w_nominal_tracking: float, q_nominal_joints: ndarray | None, init_t: bool, n_iter: int, obj_pos: ndarray, obj_quat: ndarray, target_root_pos: ndarray | None = None, object_name: str | None = None) -> tuple[ndarray, float]

Source code in eden/extensions/retargeting/omniretarget/retargeter.py

def iterate(
    self,
    q_n: np.ndarray,
    q_t_last: np.ndarray,
    target_laplacian: np.ndarray,
    adj_list: list[set],
    obj_pts_local: np.ndarray,
    foot_sticking: dict[str, bool],
    w_nominal_tracking: float,
    q_nominal_joints: np.ndarray | None,
    init_t: bool,
    n_iter: int,
    obj_pos: np.ndarray,
    obj_quat: np.ndarray,
    target_root_pos: np.ndarray | None = None,
    object_name: str | None = None,
) -> tuple[np.ndarray, float]:
    # Pre-compute foot positions at q_t_last (for sticking constraint)
    p_foot_t_last = self._get_link_positions(q_t_last, self.foot_links)

    last_cost = np.inf
    for _ in range(n_iter):
        q_n, cost = self.solve_single_iteration(
            q_n=q_n,
            q_t_last=q_t_last,
            target_laplacian=target_laplacian,
            adj_list=adj_list,
            obj_pts_local=obj_pts_local,
            foot_sticking=foot_sticking,
            w_nominal_tracking=w_nominal_tracking,
            q_nominal_joints=q_nominal_joints,
            init_t=init_t,
            obj_pos=obj_pos,
            obj_quat=obj_quat,
            target_root_pos=target_root_pos,
            p_foot_t_last=p_foot_t_last,
            object_name=object_name,
        )
        if np.abs(cost - last_cost) < 1e-4:
            break
        last_cost = cost

    return q_n, cost

solve_single_iteration

solve_single_iteration(q_n: ndarray, q_t_last: ndarray, target_laplacian: ndarray, adj_list: list[set], obj_pts_local: ndarray, foot_sticking: dict[str, bool], w_nominal_tracking: float, q_nominal_joints: ndarray | None, init_t: bool, obj_pos: ndarray, obj_quat: ndarray, target_root_pos: ndarray | None = None, p_foot_t_last: dict[str, ndarray] | None = None, object_name: str | None = None) -> tuple[ndarray, float]

Build & solve a single convex sub-problem.

Returns (q_new, cost).

Source code in eden/extensions/retargeting/omniretarget/retargeter.py

def solve_single_iteration(
    self,
    q_n: np.ndarray,
    q_t_last: np.ndarray,
    target_laplacian: np.ndarray,
    adj_list: list[set],
    obj_pts_local: np.ndarray,
    foot_sticking: dict[str, bool],
    w_nominal_tracking: float,
    q_nominal_joints: np.ndarray | None,
    init_t: bool,
    obj_pos: np.ndarray,
    obj_quat: np.ndarray,
    target_root_pos: np.ndarray | None = None,
    p_foot_t_last: dict[str, np.ndarray] | None = None,
    object_name: str | None = None,
) -> tuple[np.ndarray, float]:
    """Build & solve a single convex sub-problem.

    Returns ``(q_new, cost)``.
    """
    # ---- 1. Set current configuration ------------------------------------
    q = q_n
    self._set_entity_qpos(q)

    # ---- 2. Jacobians & positions for Laplacian links --------------------
    in_obj_frame = object_name is not None
    J_dict, p_dict = self._get_manipulator_jacobians(
        self.laplacian_match_links,
        obj_frame=in_obj_frame,
        obj_pos=obj_pos,
        obj_quat=obj_quat,
    )

    robot_link_keys = list(self.laplacian_match_links.keys())
    V_r = len(robot_link_keys)
    V_o = len(obj_pts_local)
    V = V_r + V_o

    # Stack Jacobians: robot links have non-zero J, object pts have zero J
    J_V = np.zeros((3 * V, self.nq_opt))
    for idx, key in enumerate(robot_link_keys):
        J_V[3 * idx : 3 * (idx + 1), :] = J_dict[key]
    # Object points: rows stay zero (object doesn't move with robot DOFs)

    robot_pts = np.array([p_dict[k] for k in robot_link_keys])  # (V_r, 3)
    vertices = np.vstack([robot_pts, obj_pts_local])  # (V, 3)

    # ---- 3. Laplacian constraint -----------------------------------------
    L = calculate_laplacian_matrix(vertices, adj_list)
    Kron = sp.kron(L, sp.eye(3, format="csr"), format="csr")
    J_L = Kron @ J_V  # (3V, nq_opt)

    lap0 = L @ vertices
    lap0_vec = lap0.reshape(-1)
    target_lap_vec = target_laplacian.reshape(-1)

    w_v = self.laplacian_weights * np.ones(V)
    sqrt_w3 = np.sqrt(np.repeat(w_v, 3))

    # ---- 4. CVXPY variables & constraints --------------------------------
    dqa = cp.Variable(self.nq_opt, name="dqa")
    lap_var = cp.Variable(3 * V, name="laplacian")

    constraints = [
        cp.Constant(J_L) @ dqa - lap_var == -lap0_vec,
    ]

    # ---- 4a. Joint limits ------------------------------------------------
    if self.activate_joint_limits:
        j_start = self._joint_dof_start
        j_end = j_start + self.n_joints
        joints_dqa = dqa[j_start:j_end]
        joints_curr = q[self._joint_q_start :]
        joints_lb = self.dqa_lb[j_start:j_end]
        joints_ub = self.dqa_ub[j_start:j_end]
        constraints.append(joints_dqa >= joints_lb - joints_curr)
        constraints.append(joints_dqa <= joints_ub - joints_curr)

    # ---- 4b. Trust region (SOC) ------------------------------------------
    soc_constraint = cp.SOC(self.step_size, dqa)
    constraints.append(soc_constraint)

    # ---- 4c. Foot sticking -----------------------------------------------
    if self.activate_foot_sticking and self.foot_sticking_map:
        for sticking_key, foot_link_keys in self.foot_sticking_map.items():
            if not foot_sticking.get(sticking_key, False):
                continue
            for key in foot_link_keys:
                link_name = self.foot_links[key]
                J_foot, p_foot = self._get_link_jacobian_and_pos(link_name, offset=self.link_offsets.get(key))
                if p_foot_t_last is not None and key in p_foot_t_last:
                    p_prev = p_foot_t_last[key]
                else:
                    p_prev = p_foot

                # Constrain XY displacement: foot should stay put
                # J_xy @ dqa ≈ (p_prev - p_curr)_xy ± tol
                diff_xy = p_prev[:2] - p_foot[:2]
                J_xy = J_foot[:2, :]
                tol = self.foot_sticking_tolerance
                constraints.append(J_xy @ dqa >= diff_xy - tol)
                constraints.append(J_xy @ dqa <= diff_xy + tol)

                # Optional Z sticking during contact.
                if self.activate_foot_z_sticking:
                    J_z = J_foot[2, :]
                    z_ref = p_prev[2]
                    if self.activate_foot_ground_constraint:
                        z_ref = max(z_ref, self.foot_ground_z + self.foot_ground_margin)
                    diff_z = z_ref - p_foot[2]
                    z_tol = self.foot_z_sticking_tolerance
                    constraints.append(J_z @ dqa >= diff_z - z_tol)
                    constraints.append(J_z @ dqa <= diff_z + z_tol)

    # ---- 4d. Foot-ground non-penetration (world-frame) -------------------
    if self.activate_foot_ground_constraint and self.foot_links:
        z_lb = self.foot_ground_z + self.foot_ground_margin
        for key, link_name in self.foot_links.items():
            J_foot, p_foot = self._get_link_jacobian_and_pos(link_name, offset=self.link_offsets.get(key))
            J_z = J_foot[2, :]
            constraints.append(J_z @ dqa >= z_lb - p_foot[2])

    # ---- 4e. Object non-penetration (point-based) ------------------------
    if self.activate_obj_non_penetration and V_o > 0 and V_r > 0:
        # robot_pts and obj_pts_local are BOTH in the object frame
        # (when object_name is not None) or both in world frame
        # (when object_name is None), so distances are consistent.
        diff = robot_pts[:, None, :] - obj_pts_local[None, :, :]  # (Vr, Vo, 3)
        dists = np.linalg.norm(diff, axis=-1)  # (Vr, Vo)

        close_mask = dists < self.collision_detection_threshold
        close_indices = np.argwhere(close_mask)

        for r_idx, o_idx in close_indices:
            dist = dists[r_idx, o_idx]
            if dist < 1e-6:
                continue
            normal = diff[r_idx, o_idx] / dist
            J_r = J_V[3 * r_idx : 3 * (r_idx + 1), :]
            constraints.append(normal @ (J_r @ dqa) >= self.penetration_tolerance - dist)

    # ---- 5. Objective ----------------------------------------------------
    obj_terms = []

    # Laplacian deformation
    obj_terms.append(cp.sum_squares(cp.multiply(sqrt_w3, lap_var - target_lap_vec)))

    # Root position tracking
    # p_dict and J_dict are already in the correct frame (object frame
    # when object_name is not None, world frame otherwise).
    # target_root_pos is also in the same frame (human_in_obj[0] when
    # object, raw world position when ground).
    if target_root_pos is not None and V_r > 0 and not self.entity.has_immobile_base:
        root_pos = p_dict[robot_link_keys[0]]
        root_J = J_dict[robot_link_keys[0]]

        pos_err = root_pos - target_root_pos
        obj_terms.append(self.w_root * cp.sum_squares(root_J @ dqa + pos_err))

    # Nominal joint tracking
    if w_nominal_tracking > 0 and q_nominal_joints is not None and len(self.nominal_tracking_joint_indices) > 0:
        idx_j = self.nominal_tracking_joint_indices  # into joint array
        idx_dqa = self.nominal_tracking_dof_indices  # into dqa
        q_curr_joints = q[self._joint_q_start :]
        z = dqa[idx_dqa] - (q_nominal_joints[idx_j] - q_curr_joints[idx_j])
        obj_terms.append(w_nominal_tracking * cp.sum_squares(z))

    # Per-joint regularisation
    Q_sqrt = np.sqrt(np.asarray(self.Q_diag, dtype=float))
    q_dof = self._qpos_to_dof_values(q)
    obj_terms.append(cp.sum_squares(cp.multiply(Q_sqrt, dqa + q_dof)))

    # Temporal smoothness: pull q + dqa toward the previous frame's solution
    dqa_smooth = self._qpos_to_dof_values(q_t_last) - q_dof
    obj_terms.append(self.smooth_weight * cp.sum_squares(dqa - dqa_smooth))

    # ---- 6. Solve --------------------------------------------------------
    problem = cp.Problem(cp.Minimize(cp.sum(obj_terms)), constraints)

    try:
        problem.solve(solver=cp.CLARABEL)
    except cp.error.SolverError:
        problem.solve(solver=cp.SCS)

    # Fallback: drop SOC trust region on first frame if infeasible
    if problem.status not in (cp.OPTIMAL, cp.OPTIMAL_INACCURATE) and init_t:
        constraints = [c for c in constraints if c is not soc_constraint]
        problem = cp.Problem(cp.Minimize(cp.sum(obj_terms)), constraints)
        try:
            problem.solve(solver=cp.CLARABEL)
        except cp.error.SolverError:
            problem.solve(solver=cp.SCS)

    if problem.status not in (cp.OPTIMAL, cp.OPTIMAL_INACCURATE):
        en.logger.warning(f"QP solve failed: {problem.status}")
        return q, float("inf")

    dqa_val = dqa.value
    q_new = self._integrate_dqa(q, dqa_val)

    return q_new, problem.value

Functions:

G1 (29-DOF) OmniRetarget setup + human-motion preprocessing.

Single source of truth shared by the eden retarget omniretarget CLI and examples/recipes/retargeting/omniretarget/retarget_robot_only.py: the foot-sticking / link-offset / manual-bound tables that configure :class:OmniRetargeter for the Unitree G1, plus the two motion-preprocessing helpers (height/scale normalization and foot-sticking detection).

Attributes

FOOT_STICKING_LINKS module-attribute

FOOT_STICKING_LINKS = {'left_sphere_1': 'left_ankle_roll_link', 'left_sphere_2': 'left_ankle_roll_link', 'left_sphere_3': 'left_ankle_roll_link', 'left_sphere_4': 'left_ankle_roll_link', 'right_sphere_1': 'right_ankle_roll_link', 'right_sphere_2': 'right_ankle_roll_link', 'right_sphere_3': 'right_ankle_roll_link', 'right_sphere_4': 'right_ankle_roll_link'}

LINK_OFFSETS module-attribute

LINK_OFFSETS = {'L_Toe': np.array([0.14, 0.0, -0.03]), 'R_Toe': np.array([0.14, 0.0, -0.03]), 'left_sphere_1': np.array([-0.05, 0.025, -0.03]), 'left_sphere_2': np.array([-0.05, -0.025, -0.03]), 'left_sphere_3': np.array([0.12, 0.03, -0.03]), 'left_sphere_4': np.array([0.12, -0.03, -0.03]), 'right_sphere_1': np.array([-0.05, 0.025, -0.03]), 'right_sphere_2': np.array([-0.05, -0.025, -0.03]), 'right_sphere_3': np.array([0.12, 0.03, -0.03]), 'right_sphere_4': np.array([0.12, -0.03, -0.03])}

MANUAL_LB module-attribute

MANUAL_LB = {'waist_roll_joint': -0.3, 'waist_pitch_joint': -0.1, 'left_wrist_roll_joint': -0.1, 'left_wrist_pitch_joint': -0.1, 'left_wrist_yaw_joint': -0.05, 'right_wrist_roll_joint': -0.1, 'right_wrist_pitch_joint': -0.1, 'right_wrist_yaw_joint': -0.05}

MANUAL_UB module-attribute

MANUAL_UB = {'waist_roll_joint': 0.3, 'left_elbow_joint': 1.4, 'left_wrist_roll_joint': 0.2, 'left_wrist_pitch_joint': 0.3, 'left_wrist_yaw_joint': 0.05, 'right_elbow_joint': 1.4, 'right_wrist_roll_joint': 0.2, 'right_wrist_pitch_joint': 0.3, 'right_wrist_yaw_joint': 0.05}

MANUAL_COST module-attribute

MANUAL_COST = {'waist_yaw_joint': 0.2, 'waist_roll_joint': 0.2}

NOMINAL_TRACKING_DOFS module-attribute

NOMINAL_TRACKING_DOFS = ['left_hip_pitch_joint', 'left_hip_roll_joint', 'left_hip_yaw_joint', 'left_knee_joint', 'left_ankle_pitch_joint', 'left_ankle_roll_joint', 'right_hip_pitch_joint', 'right_hip_roll_joint', 'right_hip_yaw_joint', 'right_knee_joint', 'right_ankle_pitch_joint', 'right_ankle_roll_joint']

FOOT_STICKING_MAP module-attribute

FOOT_STICKING_MAP = {'L_Toe': ['left_sphere_1', 'left_sphere_2', 'left_sphere_3', 'left_sphere_4'], 'R_Toe': ['right_sphere_1', 'right_sphere_2', 'right_sphere_3', 'right_sphere_4']}

Functions:

preprocess_human_motion

preprocess_human_motion(human_joints: ndarray, demo_joints: list[str], toe_names: list[str], scale: float, mat_height: float = 0.1) -> ndarray

Lower the human motion so its lowest toe sits at mat_height (at 0 when it starts below mat_height), then scale every coordinate by scale to robot size.

Returns a new array; human_joints is left as it was.

Source code in eden/extensions/retargeting/omniretarget/g1_presets.py

def preprocess_human_motion(
    human_joints: np.ndarray,
    demo_joints: list[str],
    toe_names: list[str],
    scale: float,
    mat_height: float = 0.1,
) -> np.ndarray:
    """Lower the human motion so its lowest toe sits at ``mat_height`` (at 0 when it starts below
    ``mat_height``), then scale every coordinate by ``scale`` to robot size.

    Returns a new array; ``human_joints`` is left as it was.
    """
    toe_indices = [demo_joints.index(toe_names[0]), demo_joints.index(toe_names[1])]
    z_min = human_joints[:, toe_indices, 2].min()
    if z_min >= mat_height:
        z_min -= mat_height
    lowered = human_joints.copy()
    lowered[:, :, 2] -= z_min
    return lowered * scale

extract_foot_sticking_velocity

extract_foot_sticking_velocity(human_joints: ndarray, demo_joints: list[str], toe_names: list[str], velocity_threshold: float = 0.01) -> list[dict[str, bool]]

Flag, per frame, whether each toe's planar displacement since the previous frame is at most velocity_threshold (metres per frame, not m/s). Frame 0 is never flagged.

Source code in eden/extensions/retargeting/omniretarget/g1_presets.py

def extract_foot_sticking_velocity(
    human_joints: np.ndarray,
    demo_joints: list[str],
    toe_names: list[str],
    velocity_threshold: float = 0.01,
) -> list[dict[str, bool]]:
    """Flag, per frame, whether each toe's planar displacement since the previous frame is at most
    ``velocity_threshold`` (metres per frame, not m/s). Frame 0 is never flagged.
    """
    left_idx = demo_joints.index(toe_names[0])
    right_idx = demo_joints.index(toe_names[1])
    left_vel = np.linalg.norm(np.diff(human_joints[:, left_idx, :2], axis=0), axis=1)
    right_vel = np.linalg.norm(np.diff(human_joints[:, right_idx, :2], axis=0), axis=1)
    left_vel = np.concatenate([[velocity_threshold + 1], left_vel])
    right_vel = np.concatenate([[velocity_threshold + 1], right_vel])
    return [
        {toe_names[0]: left_vel[i] <= velocity_threshold, toe_names[1]: right_vel[i] <= velocity_threshold}
        for i in range(len(human_joints))
    ]

Interaction-mesh helpers for OmniRetarget: Delaunay adjacency and Laplacian coordinates.

Functions:

compute_delaunay_simplices

compute_delaunay_simplices(vertices: ndarray) -> ndarray

Compute Delaunay tetrahedralization, returning simplex indices.

Source code in eden/extensions/retargeting/omniretarget/utils.py

def compute_delaunay_simplices(vertices: np.ndarray) -> np.ndarray:
    """Compute Delaunay tetrahedralization, returning simplex indices."""
    return Delaunay(vertices).simplices

get_adjacency_list

get_adjacency_list(tetrahedra: ndarray, num_vertices: int) -> list[set]

Build an adjacency list from tetrahedra.

Returns a list of sets (one per vertex) containing neighbour indices.

Source code in eden/extensions/retargeting/omniretarget/utils.py

def get_adjacency_list(tetrahedra: np.ndarray, num_vertices: int) -> list[set]:
    """Build an adjacency list from tetrahedra.

    Returns a list of *sets* (one per vertex) containing neighbour indices.
    """
    adj: list[set] = [set() for _ in range(num_vertices)]
    for tet in tetrahedra:
        for i in range(4):
            for j in range(i + 1, 4):
                u, v = tet[i], tet[j]
                adj[u].add(v)
                adj[v].add(u)
    return adj

calculate_laplacian_matrix

calculate_laplacian_matrix(vertices: ndarray, adj_list: list[set]) -> csr_matrix

Compute the (N x N) Laplacian matrix (uniform weights, sparse CSR) of vertices.

adj_list holds each vertex's neighbours, e.g. from :func:get_adjacency_list.

Source code in eden/extensions/retargeting/omniretarget/utils.py

def calculate_laplacian_matrix(vertices: np.ndarray, adj_list: list[set]) -> sp.csr_matrix:
    """Compute the (N x N) Laplacian matrix (uniform weights, sparse CSR) of ``vertices``.

    ``adj_list`` holds each vertex's neighbours, e.g. from :func:`get_adjacency_list`.
    """
    rows, cols, data = [], [], []
    for i, neighbors in enumerate(adj_list):
        for n in neighbors:
            rows.append(i)
            cols.append(n)
            data.append(1)
    adjacency_matrix = sp.coo_matrix((data, (rows, cols)), shape=(len(vertices), len(vertices)))

    degrees = np.array(adjacency_matrix.sum(axis=1)).flatten()
    degrees[degrees == 0] = 1.0
    inv_degree = sp.diags(1.0 / degrees)

    return sp.eye(len(vertices)) - inv_degree @ adjacency_matrix

calculate_laplacian_coordinates

calculate_laplacian_coordinates(vertices: ndarray, adj_list: list[set]) -> tuple[csr_matrix, ndarray]

Return (laplacian_matrix, laplacian_coordinates).

Laplacian coordinates = L @ V, capturing per-vertex differential structure of the mesh.

Source code in eden/extensions/retargeting/omniretarget/utils.py

def calculate_laplacian_coordinates(vertices: np.ndarray, adj_list: list[set]) -> tuple[sp.csr_matrix, np.ndarray]:
    """Return ``(laplacian_matrix, laplacian_coordinates)``.

    Laplacian coordinates = L @ V, capturing per-vertex differential
    structure of the mesh.
    """
    L = calculate_laplacian_matrix(vertices, adj_list)
    return L, L @ vertices

OTRetarget

Joint robot and object retargeting from surface interactions via entropic optimal transport. OTRetargeter is the solver; FloorStance and HandGrasp are its add-ons; the presets bundle G1 and LimX Oli rigs, with or without Wuji hands.

OTRetarget: joint robot and multi-object motion retargeting via optimal transport (arXiv 2609.36602).

Per frame t the retargeter solves (Eq. 1)::

min_{q_t, {T^o_t}}  E_int + E_sty_r + E_sty_o + |q_t (-) q_{t-1}|^2      s.t.  C(q_t, {T^o_t})

• E_int (Eq. 5-6) carries the demonstration's surface interactions. Every probe p (dense surface samples on the robot links and on the objects) reads a proximity triple (d, w, n) from every channel c (the ground and each object): signed distance, witness (closest surface point) and the witness-to-probe direction. Its target triple comes from the human probe it is transported from (src(p), one entropic OT plan per pre-assigned part/link group, solved in the T-pose) or, for an object probe, from the same point of the demonstrated object. Residuals penalise the depth gap, the geodesic witness displacement and the wrong side of the demonstrated contact plane, weighted by proximity kernels and normalised per link.
• E_sty_r / E_sty_o (Eq. 7) keep the motion style: tracked links follow the human joint orientations through T-pose alignment offsets, and positions derived from those orientations by forward kinematics over the robot's own link lengths (native scale — nothing is rescaled); each object is centred on its demonstrated trajectory.
• C holds joint position and velocity limits, signed-distance non-penetration of every probe-channel pair, capsule self-collision, and per-object displacement/corridor boxes.

The robot pose and every free object pose are solved jointly by a trust-region SQP (Eq. 8): Gauss-Newton Hessian, linearised constraints, one dense QP per iterate (ProxQP, :mod:.solver), 50 iterations on the cold-started first frame and a few warm-started ones after.

This module is the method alone; it knows nothing of feet or hands. What one kind of embodiment or data needs on top is an :class:Addon (OTRetargeter.addons): :class:~.stance.FloorStance (legged robots on a floor) and :class:~.grasp.HandGrasp (dexterous hands). examples/recipes/retargeting/otretarget/README.md tabulates every departure from the paper, with its measurements.

Classes

InteractionObject dataclass

InteractionObject(name: str, mesh: Trimesh, poses: ndarray, demo_mesh: Trimesh | None = None, fixed: bool = False, weight: float = 1.0)

A manipulated (or static) object of the demonstration.

ATTRIBUTE	DESCRIPTION
name	Channel name (used for the output poses). TYPE: str
mesh	The object geometry in the retargeted scene, in the object frame. TYPE: Trimesh
poses	(T, 7) demonstrated poses [pos, quat_wxyz] of the object frame. TYPE: ndarray
demo_mesh	The geometry the human actually interacted with, when mesh is a substitute (Sec. III-E). Demonstrated witnesses are mapped onto mesh by an object-to-object transport plan; distances keep their metric value. TYPE: (Trimesh, optional)
fixed	Hold the object on its demonstrated trajectory instead of making its pose a decision variable. TYPE: bool
weight	The object's f_l (default 1): multiplies the interaction weights of its probes, as link_weights does for a robot link. It belongs to the object, so a retargeter reused over clips needs no per-clip option. TYPE: float

Attributes

name instance-attribute

name: str

mesh instance-attribute

mesh: Trimesh

poses instance-attribute

poses: ndarray

demo_mesh class-attribute instance-attribute

demo_mesh: Trimesh | None = None

fixed class-attribute instance-attribute

fixed: bool = False

weight class-attribute instance-attribute

weight: float = 1.0

Methods:

RetargetResult dataclass

RetargetResult(qpos: ndarray, entity_qpos: dict[str, ndarray], object_poses: dict[str, ndarray], fps: float, info: dict[str, ndarray] = dict(), stats: dict[str, float] = dict())

Retargeted trajectories.

ATTRIBUTE	DESCRIPTION
qpos	(T, n_q) rig configuration (root qpos [pos, quat_wxyz, joints], then each welded entity's joints). TYPE: ndarray
entity_qpos	The same split per entity, ready for entity.set_qpos. TYPE: dict[str, ndarray]
object_poses	(T, 7) solved [pos, quat_wxyz] per object. TYPE: dict[str, ndarray]
fps	Frame rate. TYPE: float
info	Per-frame solver diagnostics (iterations, cost and its terms cost/<term> at the last linearisation, active_pairs, solve_ms, the number of iterates whose QP needed the elastic relaxation, qp_relaxed, or found no step at all, qp_failed, the trust-region steps rejected for deepening a penetration, steps_rejected, and the worst probe penetration at the solution, penetration_mm), plus what the add-ons report. TYPE: dict[str, ndarray]
stats	Whole-run figures: setup_s (probing, transport plans, channel grids), solve_s (the frame loop), total_s, the problem size (robot_probes, human_probes, object_probes, channels, capsule_pairs, n_dof, n_decision) and frames. TYPE: dict[str, float]

Attributes

qpos instance-attribute

qpos: ndarray

entity_qpos instance-attribute

entity_qpos: dict[str, ndarray]

object_poses instance-attribute

object_poses: dict[str, ndarray]

fps instance-attribute

fps: float

info class-attribute instance-attribute

info: dict[str, ndarray] = field(default_factory=dict)

stats class-attribute instance-attribute

stats: dict[str, float] = field(default_factory=dict)

Methods:

save

save(path: str) -> None

Write everything to one .npz.

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def save(self, path: str) -> None:
    """Write everything to one ``.npz``."""
    arrays = {"qpos": self.qpos, "fps": np.float64(self.fps)}
    arrays |= {f"entity_qpos/{k}": v for k, v in self.entity_qpos.items()}
    arrays |= {f"object_poses/{k}": v for k, v in self.object_poses.items()}
    arrays |= {f"info/{k}": v for k, v in self.info.items()}
    arrays |= {f"stats/{k}": np.float64(v) for k, v in self.stats.items()}
    np.savez_compressed(path, **arrays)

Targets dataclass

Targets(distance: ndarray, witness: ndarray, normal: ndarray, geodesic_hat: list | None = None, closed: ndarray | None = None)

Demonstrated triples for one probe set against one channel (channel frame).

Attributes

distance instance-attribute

distance: ndarray

witness instance-attribute

witness: ndarray

normal instance-attribute

normal: ndarray

geodesic_hat class-attribute instance-attribute

geodesic_hat: list | None = field(default=None, repr=False)

closed class-attribute instance-attribute

closed: ndarray | None = field(default=None, repr=False)

Methods:

Frame dataclass

Frame(t: int, q_prev: ndarray | None, obj_prev_pos: ndarray, obj_prev_rot: ndarray, demo_pos: ndarray, demo_rot: ndarray, demo_step: ndarray | None, demo_delta: ndarray | None, root_delta: ndarray | None, robot_t: list[Targets], object_t: list[Targets], human_t: list[Targets], style_pos: ndarray, style_rot: ndarray, dt: float, free_cols: ndarray | None = None, stage: str = 'main', extra: dict = dict())

Targets and bounds of one frame's solve. q_prev and the previous object poses are what it is smoothed toward (q_prev None: not smoothed); free_cols (velocity columns) are the only ones that move when set, and stage names the solve ("main", or an add-on's stage). extra holds what add-ons put in for their terms.

Attributes

t instance-attribute

t: int

q_prev instance-attribute

q_prev: ndarray | None

obj_prev_pos instance-attribute

obj_prev_pos: ndarray

obj_prev_rot instance-attribute

obj_prev_rot: ndarray

demo_pos instance-attribute

demo_pos: ndarray

demo_rot instance-attribute

demo_rot: ndarray

demo_step instance-attribute

demo_step: ndarray | None

demo_delta instance-attribute

demo_delta: ndarray | None

root_delta instance-attribute

root_delta: ndarray | None

robot_t instance-attribute

robot_t: list[Targets]

object_t instance-attribute

object_t: list[Targets]

human_t instance-attribute

human_t: list[Targets]

style_pos instance-attribute

style_pos: ndarray

style_rot instance-attribute

style_rot: ndarray

dt instance-attribute

dt: float

free_cols class-attribute instance-attribute

free_cols: ndarray | None = None

stage class-attribute instance-attribute

stage: str = 'main'

extra class-attribute instance-attribute

extra: dict = field(default_factory=dict)

Methods:

ProbeSet dataclass

ProbeSet(kind: str, ci: int, x: ndarray, d: ndarray, grad: ndarray, wit: ndarray, wjac: ndarray, targets: Targets, active: ndarray, collide: ndarray, world: ndarray, R: ndarray, tau: ndarray)

One probe set (robot or object probes) against one channel at a linearisation point.

Attributes

kind instance-attribute

kind: str

ci instance-attribute

ci: int

x instance-attribute

x: ndarray

d instance-attribute

d: ndarray

grad instance-attribute

grad: ndarray

wit instance-attribute

wit: ndarray

wjac instance-attribute

wjac: ndarray

targets instance-attribute

targets: Targets

active instance-attribute

active: ndarray

collide instance-attribute

collide: ndarray

world instance-attribute

world: ndarray

R instance-attribute

R: ndarray

tau instance-attribute

tau: ndarray

Methods:

Linearization dataclass

Linearization(frame: Frame, gn: GaussNewton, obj_pos: ndarray, obj_rot: ndarray, link_pos: ndarray, link_rot: ndarray, link_jac: ndarray, r_world: ndarray, support_C: list, support_l: list)

What an add-on's terms see while :meth:OTRetargeter._linearize builds one QP.

Attributes

frame instance-attribute

frame: Frame

gn instance-attribute

gn: GaussNewton

obj_pos instance-attribute

obj_pos: ndarray

obj_rot instance-attribute

obj_rot: ndarray

link_pos instance-attribute

link_pos: ndarray

link_rot instance-attribute

link_rot: ndarray

link_jac instance-attribute

link_jac: ndarray

r_world instance-attribute

r_world: ndarray

support_C instance-attribute

support_C: list

support_l instance-attribute

support_l: list

Methods:

Run dataclass

Run(human: HumanSurfaceMotion, objects: list[InteractionObject], out_q: ndarray, out_obj: ndarray, info: dict[str, ndarray])

One retarget_motion call: the demonstration and the outputs written so far.

Attributes

human instance-attribute

human: HumanSurfaceMotion

objects instance-attribute

objects: list[InteractionObject]

out_q instance-attribute

out_q: ndarray

out_obj instance-attribute

out_obj: ndarray

info instance-attribute

info: dict[str, ndarray]

Methods:

Addon

Addon(rt: OTRetargeter, options)

Bases: ConfigurableMixin[DynamicConfigurableOptions]

A stage or cost term added to :class:OTRetargeter for one kind of embodiment or data (see addons).

The retargeter calls each hook at a fixed point of its loop; the defaults do nothing. Options are the add-on's class attributes, as for any configurable; an unknown key is an error at construction.

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def __init__(self, rt: OTRetargeter, options) -> None:
    super().__init__(options)
    self.rt = rt

Attributes

rt instance-attribute

rt = rt

Methods:

setup

setup(human: HumanSurfaceMotion) -> None

Per demonstration, after the transport plans and the object channels; resets per-clip state.

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def setup(self, human: HumanSurfaceMotion) -> None:
    """Per demonstration, after the transport plans and the object channels; resets per-clip state."""

cold_starts

cold_starts(human: HumanSurfaceMotion, obj_pos: ndarray, obj_rot: ndarray) -> list[ndarray]

Further cold-start configurations of frame 0 (each is solved; rank_start helps pick one).

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def cold_starts(self, human: HumanSurfaceMotion, obj_pos: np.ndarray, obj_rot: np.ndarray) -> list[np.ndarray]:
    """Further cold-start configurations of frame 0 (each is solved; ``rank_start`` helps pick one)."""
    return []

rank_start

rank_start(human, frame: Frame, q, obj_pos, obj_rot, n_starts: int) -> float

Rank of one frame-0 solve among n_starts (lower wins; ties go to fewer relaxed QPs, then cost).

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def rank_start(self, human, frame: Frame, q, obj_pos, obj_rot, n_starts: int) -> float:
    """Rank of one frame-0 solve among ``n_starts`` (lower wins; ties go to fewer relaxed QPs, then cost)."""
    return 0.0

frame

frame(human: HumanSurfaceMotion, frame: Frame, forward: bool) -> None

Adjust a frame's targets, or add to frame.extra. forward is True once per frame, in time order.

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def frame(self, human: HumanSurfaceMotion, frame: Frame, forward: bool) -> None:
    """Adjust a frame's targets, or add to ``frame.extra``. ``forward`` is True once per frame, in time order."""

set_terms

set_terms(lin: Linearization, s: ProbeSet) -> None

Residuals or rows on one probe set (called after its depth residual).

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def set_terms(self, lin: Linearization, s: ProbeSet) -> None:
    """Residuals or rows on one probe set (called after its depth residual)."""

terms

terms(lin: Linearization) -> None

Residuals or rows on the whole rig (called after the style terms).

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def terms(self, lin: Linearization) -> None:
    """Residuals or rows on the whole rig (called after the style terms)."""

after_frame

after_frame(run: Run, frame: Frame, q, obj_pos, obj_rot, penetration: float)

After frame frame.t is solved: returns (q to continue from, q written out, penetration).

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def after_frame(self, run: Run, frame: Frame, q, obj_pos, obj_rot, penetration: float):
    """After frame ``frame.t`` is solved: returns ``(q to continue from, q written out, penetration)``."""
    return q, q, penetration

OTRetargeter

OTRetargeter(env: EnvBase, options: RetargeterOptions)

Bases: RetargetingBase

Optimal-transport interaction retargeter for a floating-base rig, with objects.

Rationale and measurements for the options that depart from the paper: examples/recipes/retargeting/otretarget/README.md.

PARAMETER	DESCRIPTION
hand_entity_names	Entities welded below entity_name (e.g. ["left_hand", "right_hand"]) that form one rig with it. TYPE: list[str]
correspondences	[[human parts...], [robot links...]] groups; one transport plan per group (Sec. III-B). TYPE: list
style_links	{robot link: human joint} tracked by the style cost; the first entry is the rig root. TYPE: dict[str, str]
human_root_joint	The human joint the rig root's position target sits on (the pelvis; Hips on the SOMA rig). TYPE: str
tpose	Joint values that put the robot in the human rest T-pose (unlisted joints are zero). TYPE: dict[str, float]
robot_probe_density	{name regex: probes per m^2} for robot links / human parts. TYPE: dict[str, float]
human_probe_density	{name regex: probes per m^2} for robot links / human parts. TYPE: dict[str, float]
object_probe_density	Probes per m^2 on objects (1000 in the paper). TYPE: float
min_probes_per_link	Floor on probes per link or human part. TYPE: int
probe_exclude_links	Link regexes that carry no probes. TYPE: list[str]
use_visual_mesh	Sample robot probes and fit capsules on visual meshes. TYPE: bool
object_solid_voxel	Voxel size (m) at which an object mesh whose winding-number sign is not reliable is closed into a solid before its distance field is baked (:func:~.channels.watertight_solid); 0 keeps every scan. TYPE: float
sdf_resolution	Voxel size of the channel grids (1 cm). TYPE: float
exact_band	Probes closer than this to a mesh channel read the exact mesh distance, witness and derivatives instead of the grid's (0 keeps the grid everywhere). TYPE: float
geodesic_samples	Surface samples of each object's geodesic table (all pairs: 4000 samples take 64 MB); the density is geodesic_samples / area, never below object_probe_density. TYPE: int
max_distance	d_bar: probe-channel pairs closer than this (in the demonstration or now) are active (15 cm). TYPE: float
kernel_width	sigma of the quadratic proximity kernels (13 cm). TYPE: float
kernel_cap	beta_max, the cap of the depth kernel under penetration (5). TYPE: float
ot_epsilon	Entropic regularisation of the transport plans (0.1). TYPE: float
ot_normal_weight	Weight of the normal-agreement term of the transport cost (0: the paper's position-only cost). TYPE: float
interaction_weights	(lambda_d, lambda_x, lambda_n) for robot probes ((256, 625, 2500)). TYPE: list[float]
object_interaction_scale	Multiplier of those weights for object probes (4). TYPE: float
link_weights	Per-link f_l (default 1): {regex: weight} over the rig's link names, first match wins; each regex must match a link. An object's f_l is its :attr:InteractionObject.weight. TYPE: dict[str, float]
style_pos_weight	lambda_pos and lambda_rot (the paper does not publish them). TYPE: float
style_rot_weight	lambda_pos and lambda_rot (the paper does not publish them). TYPE: float
style_link_weights	Per tracked link multiplier of both style weights (default 1): {regex: weight} over style_links keys. TYPE: dict[str, float]
object_pos_weight	lambda_pos_o, lambda_rot_o: how strongly each free object is centred on its demonstrated trajectory. TYPE: float
object_rot_weight	lambda_pos_o, lambda_rot_o: how strongly each free object is centred on its demonstrated trajectory. TYPE: float
object_smooth_weight	Weight of each free object's frame-to-frame displacement held to the demonstrated one (not in Eq. 1). TYPE: float
smooth_weight	Weight of |q_t (-) q_{t-1}|^2. TYPE: float
collision_margin	eps_col: allowed penetration (0.3 mm). TYPE: float
collision_activation	Probe-channel pairs closer than this get a non-penetration row. TYPE: float
collision_spacing	Non-penetration rows sit on one robot probe per voxel of this size per link (a link smaller than eight voxels gets an eighth of its extent instead). TYPE: float
self_collision	Add capsule self-collision rows. TYPE: bool
self_collision_activation	Capsule pairs with clearance below this get a row. TYPE: float
self_collision_exclude	[regex_a, regex_b] link-name pairs left out of self-collision (both orders). TYPE: list
capsule_radius_quantile	Radius quantile of the capsule fits. TYPE: float
joint_velocity_limit	Default joint velocity limit (rad/s or m/s). TYPE: float
joint_velocity_limits	Per-joint overrides. TYPE: dict[str, float]
joint_limits	{joint-name regex: [lower, upper]}: the range the retargeted motion may use, intersected with the joint's own limits (never wider), first match wins. TYPE: dict[str, list[float]]
root_speed_slack	(m/s) Per axis, the root's displacement stays within the demonstrator's root-joint displacement plus root_speed_slack * dt (0 disables). TYPE: float
object_corridor	b_cor (m, rad): per-axis bound on an object's deviation from its demonstrated pose. TYPE: list[float]
object_speed_slack	(m/s, rad/s) added to the demonstrated per-frame displacement to form b_env. TYPE: list[float]
trust_region	Per-DoF step boxes (Delta translation m, Delta rotation/joint rad) (5 cm, 0.1 rad). TYPE: list[float]
penetration_slack	A step that drives a probe deeper than it was by more than this (m) is re-solved (see _solve_frame). TYPE: float
witness_lateral_cap	Cap on the lateral curvature factor rho / |c| of the witness residual's Gauss-Newton model. TYPE: float
reweight_every	Accepted steps between re-evaluations of the reweighted interaction terms within a frame. TYPE: int
min_trust_scale	Smallest fraction of trust_region the step halving goes to; there the step is taken regardless. TYPE: float
first_frame_iterations	SQP budgets for the cold first frame and the warm-started rest (50, 6). TYPE: int
iterations	SQP budgets for the cold first frame and the warm-started rest (50, 6). TYPE: int
step_tolerance	Stop iterating once the step's infinity norm falls below this. TYPE: float
ground	Add the floor channel; the cold start stands the rig on it. TYPE: bool
ground_height	Floor height (m). TYPE: float
addons	{name: options} of the add-ons to run, in order: floor_stance (:class:~.stance.FloorStance), hand_grasp (:class:~.grasp.HandGrasp). TYPE: dict[str, dict]

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def __init__(self, env: EnvBase, options: RetargeterOptions) -> None:
    super().__init__(env, options)
    self._validate_scalars()
    self.kin = RigKinematics(env, [self.entity_name, *self.hand_entity_names])
    kin = self.kin
    if not kin.floating:
        raise ValueError(
            f"OTRetargeter: entity {self.entity_name!r} has a fixed base, but the method retargets the root pose "
            f"with the rest of the body. Retarget a floating-base rig."
        )
    # -- names -----------------------------------------------------------
    joints = kin.var_names[kin.n_base_v :]
    self._refuse_unknown("tpose", self.tpose, joints, "the rig's joints")
    self._refuse_unknown("joint_velocity_limits", self.joint_velocity_limits, joints, "the rig's joints")
    for pattern, bounds in self.joint_limits.items():
        if not any(re.fullmatch(pattern, n) for n in joints) or len(bounds) != 2 or not bounds[0] < bounds[1]:
            raise ValueError(
                f"OTRetargeter: joint_limits entry {pattern!r}: {bounds!r} must fully match at least one joint of the "
                f"rig ({joints}) and give [lower, upper] with lower < upper."
            )
    for i, name in enumerate(joints):
        bounds = next((b for p, b in self.joint_limits.items() if re.fullmatch(p, name)), None)
        if bounds is not None:
            kin.joint_lower[i] = max(kin.joint_lower[i], float(bounds[0]))
            kin.joint_upper[i] = min(kin.joint_upper[i], float(bounds[1]))
    for pattern in self.link_weights:
        # Robot links only (an object carries its own InteractionObject.weight): a typo here read as weight 1.
        if not any(re.fullmatch(pattern, n) for n in kin.link_names):
            raise ValueError(
                f"OTRetargeter: link_weights entry {pattern!r} fully matches none of the rig's links "
                f"({list(kin.link_names)}); fix the regex or drop the entry (weight an object with "
                f"InteractionObject(weight=...))."
            )
    for pattern in self.probe_exclude_links:
        try:
            matched = any(re.fullmatch(pattern, n) for n in kin.link_names)
        except re.error as exc:
            raise ValueError(
                f"OTRetargeter: probe_exclude_links entry {pattern!r} is not a regex ({exc})."
            ) from None
        if not matched:
            raise ValueError(
                f"OTRetargeter: probe_exclude_links entry {pattern!r} fully matches none of the rig's links "
                f"({list(kin.link_names)}); fix the regex or drop the entry."
            )
    if not self.correspondences:
        raise ValueError(
            "OTRetargeter: correspondences is empty; give at least one [[human parts], [robot links]] group."
        )
    group_links = [link for group in self.correspondences for link in self._group(group)[1]]
    self._refuse_unknown("correspondences robot links", group_links, kin.link_names, "the rig's links")
    if not self.style_links:
        raise ValueError("OTRetargeter: style_links is empty; give at least the root link -> 'pelvis'.")
    self._refuse_unknown("style_links keys", self.style_links, kin.link_names, "the rig's links")
    for pattern, value in self.style_link_weights.items():
        if not any(re.fullmatch(pattern, link) for link in self.style_links) or value < 0:
            raise ValueError(
                f"OTRetargeter: style_link_weights entry {pattern!r}: {value!r} must be >= 0 and fully match at "
                f"least one style_links key ({list(self.style_links)}); fix the regex or drop the entry."
            )
    root_link = kin.link_names[int(np.flatnonzero(kin.link_parent < 0)[0])]
    first = next(iter(self.style_links))
    if first != root_link:
        raise ValueError(
            f"OTRetargeter: the first style_links entry is {first!r}, but it must be the rig root {root_link!r}: "
            f"the style targets are forward kinematics from the root aligned to the human pelvis."
        )

    # -- reference (T-) pose ---------------------------------------------
    self._q_tpose = np.zeros(kin.n_q)
    self._q_tpose[3] = 1.0
    for name, value in self.tpose.items():
        self._q_tpose[self._q_index(name)] = value
    kin.set_configuration(self._q_tpose)
    self._tpose_link_pos, self._tpose_link_rot = kin.link_poses()

    # -- robot probes ------------------------------------------------------
    self.robot_probes = sample_rig_probes(
        kin, self.robot_probe_density, self.min_probes_per_link, self.probe_exclude_links, self.use_visual_mesh
    )
    self._probe_labels = [kin.link_names[c] for c in self.robot_probes.carrier]
    # Non-penetration rows on one probe per voxel, the voxel shrunk to an eighth of a small link's extent:
    # at 1 cm a 2-4 cm phalanx keeps a handful of rows and a thin pole slips in between them.
    local, carrier = self.robot_probes.local, self.robot_probes.carrier
    spacing = np.full(len(kin.link_names), float(self.collision_spacing))
    for li in np.unique(carrier):
        pts = local[carrier == li]
        spacing[li] = min(spacing[li], 0.125 * float((pts.max(axis=0) - pts.min(axis=0)).max()))
    self._collision_probe = _voxel_subset(local, carrier, np.maximum(spacing, 1e-3)[carrier])

    # -- self-collision capsules ------------------------------------------
    self.capsules = None
    if self.self_collision:
        links = {link.name: link for ent in kin.entities for link in ent.links}
        pts = {}
        for li, name in enumerate(kin.link_names):
            geoms = links[name].vgeoms if self.use_visual_mesh else links[name].geoms
            if len(geoms) == 0:
                continue
            mesh = get_link_mesh(links[name], use_visual_mesh=self.use_visual_mesh)
            if mesh.area > 1e-8:
                pts[li] = sample_surface(mesh, 20000.0, 64, seed=li)[0]
        q_zero = np.zeros(kin.n_q)
        q_zero[3] = 1.0
        kin.set_configuration(q_zero)
        zero_pose = kin.link_poses()
        self.capsules = build_link_capsules(
            pts,
            kin.link_parent,
            [zero_pose, (self._tpose_link_pos, self._tpose_link_rot)],
            radius_quantile=self.capsule_radius_quantile,
            link_names=list(kin.link_names),
            exclude=[tuple(pair) for pair in self.self_collision_exclude],
        )

    # -- limits ------------------------------------------------------------
    self._vel_limit = np.full(kin.n_v - kin.n_base_v, float(self.joint_velocity_limit))
    for name, value in self.joint_velocity_limits.items():
        self._vel_limit[kin.var_names.index(name) - kin.n_base_v] = value
    self._link_weight = np.array([_match_weight(n, self.link_weights) for n in kin.link_names])
    self._limb_of_link, self._joint_q, self._joint_limb = self._limbs()

    # -- add-ons -----------------------------------------------------------
    classes = _addon_classes()
    unknown = [n for n in self.addons if n not in classes]
    if unknown:
        raise ValueError(
            f"OTRetargeter: addons {unknown} are not add-ons. Available: {list(classes)}. Fix the name or drop the "
            f"entry."
        )
    self._addons: list[Addon] = [
        classes[name](self, classes[name].configure(**(cfg or {}))) for name, cfg in self.addons.items()
    ]
    self._terms = _TERMS + tuple(t for a in self._addons for t in a._terms)

Attributes

hand_entity_names class-attribute instance-attribute

hand_entity_names: list[str] = []

correspondences class-attribute instance-attribute

correspondences: list = []

style_links class-attribute instance-attribute

style_links: dict[str, str] = {}

human_root_joint class-attribute instance-attribute

human_root_joint: str = 'pelvis'

tpose class-attribute instance-attribute

tpose: dict[str, float] = {}

robot_probe_density class-attribute instance-attribute

robot_probe_density: dict[str, float] = {'.*': 2000.0}

human_probe_density class-attribute instance-attribute

human_probe_density: dict[str, float] = {'.*': 2000.0}

object_probe_density class-attribute instance-attribute

object_probe_density: float = 1000.0

min_probes_per_link class-attribute instance-attribute

min_probes_per_link: int = 24

probe_exclude_links class-attribute instance-attribute

probe_exclude_links: list[str] = []

use_visual_mesh class-attribute instance-attribute

use_visual_mesh: bool = True

object_solid_voxel class-attribute instance-attribute

object_solid_voxel: float = 0.004

sdf_resolution class-attribute instance-attribute

sdf_resolution: float = 0.01

exact_band class-attribute instance-attribute

exact_band: float = 0.01

geodesic_samples class-attribute instance-attribute

geodesic_samples: int = 4000

max_distance class-attribute instance-attribute

max_distance: float = 0.15

kernel_width class-attribute instance-attribute

kernel_width: float = 0.13

kernel_cap class-attribute instance-attribute

kernel_cap: float = 5.0

ot_epsilon class-attribute instance-attribute

ot_epsilon: float = 0.1

ot_normal_weight class-attribute instance-attribute

ot_normal_weight: float = 0.5

interaction_weights class-attribute instance-attribute

interaction_weights: list[float] = [256.0, 625.0, 2500.0]

object_interaction_scale class-attribute instance-attribute

object_interaction_scale: float = 4.0

link_weights class-attribute instance-attribute

link_weights: dict[str, float] = {}

style_pos_weight class-attribute instance-attribute

style_pos_weight: float = 0.03

style_link_weights class-attribute instance-attribute

style_link_weights: dict[str, float] = {}

style_rot_weight class-attribute instance-attribute

style_rot_weight: float = 3.0

object_pos_weight class-attribute instance-attribute

object_pos_weight: float = 10.0

object_rot_weight class-attribute instance-attribute

object_rot_weight: float = 1.0

smooth_weight class-attribute instance-attribute

smooth_weight: float = 0.5

object_smooth_weight class-attribute instance-attribute

object_smooth_weight: float = 100.0

collision_margin class-attribute instance-attribute

collision_margin: float = 0.0003

collision_activation class-attribute instance-attribute

collision_activation: float = 0.15

collision_spacing class-attribute instance-attribute

collision_spacing: float = 0.01

self_collision class-attribute instance-attribute

self_collision: bool = True

self_collision_activation class-attribute instance-attribute

self_collision_activation: float = 0.03

self_collision_exclude class-attribute instance-attribute

self_collision_exclude: list = []

capsule_radius_quantile class-attribute instance-attribute

capsule_radius_quantile: float = 0.8

joint_velocity_limit class-attribute instance-attribute

joint_velocity_limit: float = 20.0

joint_velocity_limits class-attribute instance-attribute

joint_velocity_limits: dict[str, float] = {}

joint_limits class-attribute instance-attribute

joint_limits: dict[str, list[float]] = {}

root_speed_slack class-attribute instance-attribute

root_speed_slack: float = 1.0

object_corridor class-attribute instance-attribute

object_corridor: list[float] = [0.5, 1.0]

object_speed_slack class-attribute instance-attribute

object_speed_slack: list[float] = [1.0, 3.0]

trust_region class-attribute instance-attribute

trust_region: list[float] = [0.05, 0.1]

penetration_slack class-attribute instance-attribute

penetration_slack: float = 0.001

witness_lateral_cap class-attribute instance-attribute

witness_lateral_cap: float = 4.0

reweight_every class-attribute instance-attribute

reweight_every: int = 10

min_trust_scale class-attribute instance-attribute

min_trust_scale: float = 1.0 / 64.0

first_frame_iterations class-attribute instance-attribute

first_frame_iterations: int = 50

iterations class-attribute instance-attribute

iterations: int = 6

step_tolerance class-attribute instance-attribute

step_tolerance: float = 0.0001

ground class-attribute instance-attribute

ground: bool = True

ground_height class-attribute instance-attribute

ground_height: float = 0.0

addons class-attribute instance-attribute

addons: dict[str, dict] = {}

kin instance-attribute

kin = RigKinematics(env, [self.entity_name, *self.hand_entity_names])

robot_probes instance-attribute

robot_probes = sample_rig_probes(kin, self.robot_probe_density, self.min_probes_per_link, self.probe_exclude_links, self.use_visual_mesh)

capsules instance-attribute

capsules = None

Methods:

addon

addon(name: str) -> Addon | None

The add-on name of this retargeter (see addons), or None when it is not configured.

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def addon(self, name: str) -> Addon | None:
    """The add-on ``name`` of this retargeter (see ``addons``), or None when it is not configured."""
    return next((a for n, a in zip(self.addons, self._addons) if n == name), None)

retarget_motion

retarget_motion(human: HumanSurfaceMotion, objects: list[InteractionObject] | None = None, q_init: ndarray | None = None, progress: bool = True) -> RetargetResult

Retarget a demonstration frame by frame.

PARAMETER	DESCRIPTION
human	The demonstrator's posed surface and skeleton (z-up, metres, native scale). TYPE: HumanSurfaceMotion
objects	Objects of the scene with their demonstrated poses (same frames as human). TYPE: list[InteractionObject] DEFAULT: None
q_init	(n_q,) rig configuration to start the first frame from; defaults to the cold start (_initial_configuration and the add-ons' cold_starts). TYPE: ndarray DEFAULT: None
progress	Show a progress bar. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
RetargetResult

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def retarget_motion(
    self,
    human: HumanSurfaceMotion,
    objects: list[InteractionObject] | None = None,
    q_init: np.ndarray | None = None,
    progress: bool = True,
) -> RetargetResult:
    """Retarget a demonstration frame by frame.

    Parameters
    ----------
    human : HumanSurfaceMotion
        The demonstrator's posed surface and skeleton (z-up, metres, native scale).
    objects : list[InteractionObject], optional
        Objects of the scene with their demonstrated poses (same frames as ``human``).
    q_init : np.ndarray, optional
        ``(n_q,)`` rig configuration to start the first frame from; defaults to the cold start
        (``_initial_configuration`` and the add-ons' ``cold_starts``).
    progress : bool
        Show a progress bar.

    Returns
    -------
    RetargetResult
    """
    kin = self.kin
    objects = list(objects or [])
    n_frames = human.num_frames
    if n_frames < 1 or not human.fps > 0:
        raise ValueError(
            f"OTRetargeter.retarget_motion: the demonstration has {n_frames} frames at fps={human.fps!r}; it needs "
            f"at least one frame and fps > 0 (check the slice that made it, e.g. HumanSurfaceMotion.slice past the "
            f"end)."
        )
    t0 = time.time()
    self._setup(human, objects)
    setup_s = time.time() - t0
    en.logger.info(
        f"OTRetarget setup {setup_s:.1f}s: {len(self.robot_probes)} robot probes, "
        f"{len(self.human_probes)} human probes, {len(self.object_probes)} object probes, "
        f"{len(self.channels)} channels, {0 if self.capsules is None else len(self.capsules.pairs)} capsule pairs."
    )

    obj_pos = self._demo_pos[0].copy()
    obj_rot = self._demo_rot[0].copy()
    if q_init is None:
        # One BLAS thread here too: with a pool, sums are ordered differently from run to run, and a 1e-16
        # difference in the cold start grew into 0.4 rad by frame 40 of a chair flip (runs were not reproducible).
        with single_threaded_blas():
            starts = [self._initial_configuration(human, obj_pos, obj_rot)]
            for addon in self._addons:
                starts += addon.cold_starts(human, obj_pos, obj_rot)
    else:
        q = np.asarray(q_init, dtype=np.float64).copy()
        if q.shape != (kin.n_q,):
            raise ValueError(
                f"OTRetargeter: q_init has shape {q.shape}; the rig's configuration has {kin.n_q} entries."
            )
        starts = [q]

    keys = [*_INFO, *(k for a in self._addons for k in a._info), *(f"cost/{k}" for k in self._terms)]
    run = Run(human, objects, np.zeros((n_frames, kin.n_q)), np.zeros((len(objects), n_frames, 7)), {})
    run.info = {k: np.zeros(n_frames) for k in keys}
    iterator = range(n_frames)
    if progress:
        from tqdm import tqdm

        iterator = tqdm(iterator, desc="OTRetarget")
    # Every product in an iterate is small (tens of columns): a multithreaded BLAS spends far longer
    # waking its pool than computing (6 ms vs 0.03 ms for a 100x81 Gram matrix on a 64-core node).
    t1 = time.time()
    with single_threaded_blas():
        self._run_frames(iterator, run, starts, obj_pos, obj_rot)
    solve_s = time.time() - t1
    stats = {
        "setup_s": setup_s,
        "solve_s": solve_s,
        "total_s": setup_s + solve_s,
        "frames": n_frames,
        "robot_probes": len(self.robot_probes),
        "human_probes": len(self.human_probes),
        "object_probes": len(self.object_probes),
        "channels": len(self.channels),
        "capsule_pairs": 0 if self.capsules is None else len(self.capsules.pairs),
        "n_dof": kin.n_v,
        "n_decision": self.n_x,
    }
    return RetargetResult(
        qpos=run.out_q,
        entity_qpos=dict(zip(kin.entity_names, kin.entity_qpos(run.out_q))),
        object_poses={o.name: run.out_obj[oi] for oi, o in enumerate(objects)},
        fps=human.fps,
        info=run.info,
        stats=stats,
    )

solve

solve(q, obj_pos, obj_rot, frame: Frame, n_iter: int) -> tuple

One solve of frame from q for an add-on's stage: (q, obj_pos, obj_rot, local, qp_failed).

The object poses are copied; QP counts are not reported.

Source code in eden/extensions/retargeting/otretarget/retargeter.py

def solve(self, q, obj_pos, obj_rot, frame: Frame, n_iter: int) -> tuple:
    """One solve of ``frame`` from ``q`` for an add-on's stage: ``(q, obj_pos, obj_rot, local, qp_failed)``.

    The object poses are copied; QP counts are not reported.
    """
    info = _scratch_info()
    q, obj_pos, obj_rot, local, _ = self._solve_frame(q, obj_pos.copy(), obj_rot.copy(), frame, n_iter, info, 0)
    return q, obj_pos, obj_rot, local, info["qp_failed"][0] > 0

Functions:

Embodiment presets for OTRetarget: G1 and LimX Oli, with or without Wuji hands; SMPL-X or SOMA demonstrators.

A preset states what the paper calls manual pre-assignment (Sec. III-B, D):

• correspondences — which human body parts each group of robot links is matched to; the transport plan is solved per group between the two surface clouds in the T-pose.
• style_links — the robot links whose world orientation (and FK-derived position) tracks a human joint frame, through an alignment offset measured in the T-pose.
• tpose — the robot joint values that put it in the human body model's rest T-pose (arms along the lateral axis, palms down, thumbs forward, fingers extended). It is a reference posture only, never executed, so it may lie outside a joint's limits (the G1 elbow does).

Presets name human parts with SMPL-X joints (:data:~eden.extensions.retargeting.otretarget.human.SMPLX_JOINT_NAMES); :func:for_body_model translates one to the SOMA rig (BONES-SEED), whose fingers carry an extra metacarpal joint.

Attributes

Group module-attribute

Group = tuple[tuple[str, ...], tuple[str, ...]]

G1_WUJI module-attribute

G1_WUJI = RigPreset(_WUJI_RIG, _G1_TPOSE, _g1_groups(True), _g1_style(True), 'pelvis', _WUJI_DENSITY, _HUMAN_DENSITY, _WUJI_SELF_COLLISION_EXCLUDE, _WUJI_STYLE_WEIGHTS, _WUJI_INTERACTION_WEIGHTS, _WUJI_JOINT_LIMITS, _WUJI_PALMS, _wuji_tips(), _WUJI_HAND_OPTIONS)

OLI_WUJI module-attribute

OLI_WUJI = RigPreset(_WUJI_RIG, _OLI_TPOSE, _oli_groups(True), _oli_style(True), 'pelvis', _WUJI_DENSITY, _HUMAN_DENSITY, _WUJI_SELF_COLLISION_EXCLUDE, _WUJI_STYLE_WEIGHTS, _WUJI_INTERACTION_WEIGHTS, _WUJI_JOINT_LIMITS, _WUJI_PALMS, _wuji_tips(), _WUJI_HAND_OPTIONS)

G1 module-attribute

G1 = RigPreset(('robot',), _G1_TPOSE, _g1_groups(False), _g1_style(False), 'pelvis', _BODY_DENSITY, _BODY_DENSITY)

OLI module-attribute

OLI = RigPreset(('robot',), _OLI_TPOSE, _oli_groups(False), _oli_style(False), 'pelvis', _BODY_DENSITY, _BODY_DENSITY)

PRESETS module-attribute

PRESETS: dict[str, RigPreset] = {'g1_wuji': G1_WUJI, 'oli_wuji': OLI_WUJI, 'g1': G1, 'oli': OLI}

SMPLX_TO_SOMA_PARTS module-attribute

SMPLX_TO_SOMA_PARTS = _smplx_to_soma_parts()

SMPLX_TO_SOMA_JOINT module-attribute

SMPLX_TO_SOMA_JOINT = {k: v[1] if k == 'pelvis' else v[0] for k, v in SMPLX_TO_SOMA_PARTS.items()}

Classes

RigPreset dataclass

RigPreset(entities: tuple[str, ...], tpose: dict[str, float], correspondences: tuple[Group, ...], style_links: dict[str, str], human_root: str = 'pelvis', robot_probe_density: dict[str, float] = dict(), human_probe_density: dict[str, float] = dict(), self_collision_exclude: tuple[tuple[str, str], ...] = (), style_link_weights: dict[str, float] = dict(), interaction_weights: tuple[float, float, float] | None = None, joint_limits: dict[str, tuple[float, float]] = dict(), palms: dict[str, str | list[str]] = dict(), fingertips: dict[str, str] = dict(), hand_options: dict = dict())

Everything embodiment-specific the retargeter needs (see the module docstring).

ATTRIBUTE	DESCRIPTION
entities	Scene entity names of the rig, root first. TYPE: tuple[str, ...]
tpose	Joint values of the T-pose; unlisted joints are zero. TYPE: dict[str, float]
correspondences	((human parts...), (robot links...)) groups. TYPE: tuple[Group, ...]
style_links	{robot link: human joint}; the first entry is the rig root, aligned to human_root. TYPE: dict[str, str]
human_root	The human joint the rig root is aligned with. TYPE: str
robot_probe_density	{link regex: points per m^2}. TYPE: dict[str, float]
human_probe_density	{joint regex: points per m^2}. TYPE: dict[str, float]
self_collision_exclude	Link-name regex pairs left out of capsule self-collision. TYPE: tuple[tuple[str, str], ...]
style_link_weights	Per tracked link multipliers of the style weights ({link regex: weight}). TYPE: dict[str, float]
interaction_weights	(lambda_d, lambda_x, lambda_n) for robot probes; None keeps the retargeter's (the paper's) values. TYPE: (tuple[float, float, float], optional)
joint_limits	{joint regex: (lower, upper)} the retargeted motion may use, within the joints' own limits. TYPE: dict[str, tuple[float, float]]
palms	{robot palm link: human wrist joint} (or [frame joint, joints...], see HandGrasp.palms); a rig with palms gets the hand_grasp add-on (:class:~eden.extensions.retargeting.otretarget.grasp.HandGrasp). TYPE: dict[str, str | list[str]]
fingertips	{robot fingertip link: human distal-phalanx joint}. TYPE: dict[str, str]
hand_options	Further HandGrasp options (closure, cold_start_palm_weight, ...). TYPE: dict

Attributes

entities instance-attribute

entities: tuple[str, ...]

tpose instance-attribute

tpose: dict[str, float]

correspondences instance-attribute

correspondences: tuple[Group, ...]

style_links instance-attribute

style_links: dict[str, str]

human_root class-attribute instance-attribute

human_root: str = 'pelvis'

robot_probe_density class-attribute instance-attribute

robot_probe_density: dict[str, float] = field(default_factory=dict)

human_probe_density class-attribute instance-attribute

human_probe_density: dict[str, float] = field(default_factory=dict)

self_collision_exclude class-attribute instance-attribute

self_collision_exclude: tuple[tuple[str, str], ...] = ()

style_link_weights class-attribute instance-attribute

style_link_weights: dict[str, float] = field(default_factory=dict)

interaction_weights class-attribute instance-attribute

interaction_weights: tuple[float, float, float] | None = None

joint_limits class-attribute instance-attribute

joint_limits: dict[str, tuple[float, float]] = field(default_factory=dict)

palms class-attribute instance-attribute

palms: dict[str, str | list[str]] = field(default_factory=dict)

fingertips class-attribute instance-attribute

fingertips: dict[str, str] = field(default_factory=dict)

hand_options class-attribute instance-attribute

hand_options: dict = field(default_factory=dict)

Methods:

retargeter_options

retargeter_options() -> dict

The preset as :class:~eden.extensions.retargeting.otretarget.OTRetargeter configure keywords: a legged rig on the floor (floor_stance), and its hands when it has palms (hand_grasp).

Source code in eden/extensions/retargeting/otretarget/presets.py

def retargeter_options(self) -> dict:
    """The preset as :class:`~eden.extensions.retargeting.otretarget.OTRetargeter` ``configure`` keywords: a legged
    rig on the floor (``floor_stance``), and its hands when it has palms (``hand_grasp``).
    """
    addons = {"floor_stance": {}}
    if self.palms:
        addons["hand_grasp"] = dict(palms=dict(self.palms), fingertips=dict(self.fingertips), **self.hand_options)
    options = dict(
        entity_name=self.entities[0],
        hand_entity_names=list(self.entities[1:]),
        correspondences=[[list(h), list(r)] for h, r in self.correspondences],
        style_links=dict(self.style_links),
        human_root_joint=self.human_root,
        tpose=dict(self.tpose),
        robot_probe_density=dict(self.robot_probe_density),
        human_probe_density=dict(self.human_probe_density),
        self_collision_exclude=[list(p) for p in self.self_collision_exclude],
        style_link_weights=dict(self.style_link_weights),
        joint_limits={k: list(v) for k, v in self.joint_limits.items()},
        addons=addons,
    )
    if self.interaction_weights is not None:
        options["interaction_weights"] = list(self.interaction_weights)
    return options

Functions:

for_body_model

for_body_model(preset: RigPreset, body_model: str) -> RigPreset

The preset with its human side expressed in another body model's joints.

PARAMETER	DESCRIPTION
preset	A preset in SMPL-X joint names. TYPE: RigPreset
body_model	Target body model. TYPE: ('smplx', 'soma') DEFAULT: "smplx"

Source code in eden/extensions/retargeting/otretarget/presets.py

def for_body_model(preset: RigPreset, body_model: str) -> RigPreset:
    """The preset with its human side expressed in another body model's joints.

    Parameters
    ----------
    preset : RigPreset
        A preset in SMPL-X joint names.
    body_model : {"smplx", "soma"}
        Target body model.
    """
    if body_model == "smplx":
        return preset
    if body_model != "soma":
        raise ValueError(f"for_body_model: body_model={body_model!r}; expected 'smplx' or 'soma'.")
    groups = tuple(
        (tuple(j for p in human for j in SMPLX_TO_SOMA_PARTS[p]), robot) for human, robot in preset.correspondences
    )
    style = {link: SMPLX_TO_SOMA_JOINT[joint] for link, joint in preset.style_links.items()}
    dense = any(d > 2000.0 for d in preset.human_probe_density.values())
    density = {r"(Left|Right)Hand.*": _HAND_DENSITY, r".*": 2000.0} if dense else dict(_BODY_DENSITY)
    palms = {link: list(SMPLX_TO_SOMA_PARTS[joint]) for link, joint in preset.palms.items()}  # the hand + metacarpals
    tips = {link: SMPLX_TO_SOMA_JOINT[joint] for link, joint in preset.fingertips.items()}
    return replace(
        preset,
        correspondences=groups,
        style_links=style,
        human_root="Hips",
        human_probe_density=density,
        palms=palms,
        fingertips=tips,
    )

Legged stance on the floor: an add-on of :class:~.retargeter.OTRetargeter (addons={"floor_stance": {...}}).

The paper keeps foot-floor contact in the cost only, as the floor channel's interaction terms, normalised per link and at the demonstrator's own footprints. On a robot of other proportions that floats the feet, slides them and pulls them onto the human's stance width. Three additions, each an option below (measurements in examples/recipes/retargeting/otretarget/README.md):

• Footprint offset — the floor is translation invariant, so where on it a foot lands is posture: each robot probe's floor witness is shifted sideways by how much wider than its human source it sits from the root in the T-pose.
• Support — a body unit (correspondence group) whose demonstrated sole is on the floor keeps its lowest point there, as a constraint.
• No-slip stance — a foot planted on two frames follows the human foot's planar rigid motion about its own contact (anchor probes as constraints), and its floor witnesses move with it.

Classes

FloorStance

FloorStance(rt, options)

Bases: Addon

Footprint offset, floor support and no-slip stance on the floor channel (needs ground).

PARAMETER	DESCRIPTION
footprint_offset	Shift each robot probe's demonstrated floor witness sideways by how far that probe sits from the root in the T-pose beyond where its human source sits from the pelvis (lateral, in the root frame, turned with the demonstrated heading). A probe whose source stays in floor contact keeps the offset it landed with. TYPE: bool
footprint_contact	(m) A probe whose human source is within this of the floor is in contact. TYPE: float
support_height	(m) On every body unit with a downward-facing probe whose human source is within support_height of the floor, the unit's lowest downward-facing probe may rise at most support_slack above the demonstrated height (0 height disables). TYPE: float
support_slack	(m) On every body unit with a downward-facing probe whose human source is within support_height of the floor, the unit's lowest downward-facing probe may rise at most support_slack above the demonstrated height (0 height disables). TYPE: float
stick	No-slip stance: while probes of a unit have their sources in contact on two consecutive frames, a few of its probes on the floor (anchors) follow the human foot's planar rigid motion about the robot foot's own contact, within stick_slack. After lift-off its floor witnesses return to the footprint targets over stick_release of the human foot's travel. TYPE: bool
stick_slack	(m) Horizontal displacement allowed per frame beyond the demonstrated one. TYPE: float
stick_height	(m) Anchors are probes in contact within this of the robot unit's lowest point on the previous solution. TYPE: float
stick_release	(m) Human foot travel after lift-off over which the floor witnesses return to the footprint targets. TYPE: float
stick_anchors	Anchors per unit in contact: the lowest one, then each farthest from those already chosen. TYPE: int

Source code in eden/extensions/retargeting/otretarget/stance.py

def __init__(self, rt, options) -> None:
    super().__init__(rt, options)
    if not rt.ground:
        raise ValueError(
            "FloorStance: the retargeter has no floor channel (ground=False); set ground=True or drop it."
        )
    checks = {
        "footprint_contact": self.footprint_contact >= 0,
        "support_height": self.support_height >= 0,
        "support_slack": self.support_slack >= 0,
        "stick_slack": self.stick_slack >= 0,
        "stick_height": self.stick_height > 0,
        "stick_release": self.stick_release > 0,
        "stick_anchors": self.stick_anchors >= 1,
    }
    bad = [k for k, ok in checks.items() if not ok]
    if bad:
        raise ValueError(
            f"FloorStance: {', '.join(f'{k}={getattr(self, k)!r}' for k in bad)} out of range: heights, slacks and "
            f"contact must be >= 0, stick_height and stick_release > 0, stick_anchors >= 1."
        )

Attributes

footprint_offset class-attribute instance-attribute

footprint_offset: bool = True

footprint_contact class-attribute instance-attribute

footprint_contact: float = 0.02

support_height class-attribute instance-attribute

support_height: float = 0.02

support_slack class-attribute instance-attribute

support_slack: float = 0.005

stick class-attribute instance-attribute

stick: bool = True

stick_slack class-attribute instance-attribute

stick_slack: float = 0.0002

stick_height class-attribute instance-attribute

stick_height: float = 0.005

stick_release class-attribute instance-attribute

stick_release: float = 0.1

stick_anchors class-attribute instance-attribute

stick_anchors: int = 3

Methods:

setup

setup(human: HumanSurfaceMotion) -> None

Source code in eden/extensions/retargeting/otretarget/stance.py

def setup(self, human: HumanSurfaceMotion) -> None:
    rt = self.rt
    self._ground = rt._channel_object.index(-1)
    # Footprint offsets: where each robot probe sits relative to the root in the T-pose, minus where its
    # human source sits relative to the pelvis, in the root's T-pose frame.
    root = rt._style_link_idx[0]
    R_root_t, x_root_t = rt._tpose_link_rot[root], rt._tpose_link_pos[root]
    R_hp, x_hp = human.rest_joint_rot[rt._pelvis_joint], human.rest_joint_pos[rt._pelvis_joint]
    r_tpose = rt.robot_probes.world(rt._tpose_link_pos, rt._tpose_link_rot)
    h_rest = rt.human_probes.positions(human.rest_vertices)
    src = np.maximum(rt.src, 0)
    robot_rel = (r_tpose - x_root_t) @ R_root_t
    human_rel = ((h_rest[src] - x_hp) @ R_hp) @ rt._style_offset[0]
    offset = robot_rel - human_rel
    # Lateral only: the offset is there for stance width. The fore-aft part measures where a skeleton's
    # ankle sits under its pelvis in the reference pose, which is no reason to plant a foot elsewhere.
    offset[:, [0, 2]] = 0.0
    offset[rt.src < 0] = 0.0
    self.offset = offset
    self._footprint = None  # (held floor-witness offsets, contact) of the previous frame
    self._stance = None  # (demonstrated floor distance, human source positions) of the previous frame
    self._carry = None  # (per-probe floor-witness offset carried off the floor, human travel since, planted)

frame

frame(human: HumanSurfaceMotion, frame: Frame, forward: bool) -> None

Source code in eden/extensions/retargeting/otretarget/stance.py

def frame(self, human: HumanSurfaceMotion, frame: Frame, forward: bool) -> None:
    rt = self.rt
    targets = frame.robot_t[self._ground]
    if self.footprint_offset:
        # A planted foot does not turn with the pelvis: a probe whose source is in floor contact keeps the offset
        # it landed with, and only a lifted one re-aims it at the current heading (turned every frame, a 10 cm
        # offset swept the stance foot of a spin round a 10 cm circle).
        R_root = human.joint_rot[frame.t, rt._pelvis_joint] @ rt._style_offset[0]
        heading = quat_from_axis_angle(np.array([0.0, 0.0, yaw_from_quat(R_to_quat(R_root))]))
        turned = self.offset @ quat_to_R(heading).T  # the offsets are horizontal: z stays 0
        h = frame.human_t[self._ground]
        contact = (rt.src >= 0) & (h.distance[np.maximum(rt.src, 0)] < self.footprint_contact)
        if self._footprint is not None:
            held, was_contact = self._footprint
            keep = contact & was_contact
            turned[keep] = held[keep]
        if forward:
            # Only the forward pass advances: a frame read again later (a stage re-solving earlier frames) would
            # leave the next frame comparing its contact against a frame long past.
            self._footprint = (turned, contact)
        targets.witness = targets.witness + turned
    if forward and self.stick:
        frame.extra["stick"] = self._anchors(human, frame.t, targets, frame.q_prev)

set_terms

set_terms(lin: Linearization, s: ProbeSet) -> None

Source code in eden/extensions/retargeting/otretarget/stance.py

def set_terms(self, lin: Linearization, s: ProbeSet) -> None:
    if s.kind != "robot" or s.ci != self._ground or self.support_height <= 0:
        return
    # Support: a body unit (e.g. ankle + foot links) whose demonstrated sole is on the floor keeps its lowest point
    # there, d <= (d_hat)+ + slack. One row per unit, on its lowest downward-facing probe whether that probe is the
    # supported one or not: a rigid foot cannot put every probe paired with a floor-touching human point down.
    rt = self.rt
    probes = rt.robot_probes
    down = np.einsum("nab,nb->na", lin.link_rot[probes.carrier], probes.normals)[:, 2] < -0.7
    candidates = down & (rt.src >= 0)
    supported = np.flatnonzero((s.targets.distance < self.support_height) & candidates)
    units = rt.probe_group[supported]
    for unit in np.unique(units):
        on_unit = np.flatnonzero(candidates & (rt.probe_group == unit))
        k = on_unit[np.argmin(s.d[on_unit])]
        target = max(float(s.targets.distance[supported[units == unit]].min()), 0.0)
        target += self.support_slack
        rows = rt._rows(
            "robot", s.ci, np.array([k]), s.world[[k]], s.R, s.tau, lin.obj_pos, lin.link_pos, s.grad[[k]]
        )
        lin.support_C.append(-lin.gn.dense(rows))
        lin.support_l.append(np.array([s.d[k] - target]))

terms

terms(lin: Linearization) -> None

Source code in eden/extensions/retargeting/otretarget/stance.py

def terms(self, lin: Linearization) -> None:
    stick = lin.frame.extra.get("stick")
    if stick is None:
        return
    # No-slip stance: each anchor's horizontal position stays within ``tol`` of its target.
    rt = self.rt
    idx, target_xy, tol = stick
    R, tau = rt._channel_pose(self._ground, lin.obj_pos, lin.obj_rot)
    slide = lin.r_world[idx, :2] - target_xy
    for axis in range(2):
        gamma = np.zeros((len(idx), 3))
        gamma[:, axis] = 1.0
        rows = rt._rows("robot", self._ground, idx, lin.r_world[idx], R, tau, lin.obj_pos, lin.link_pos, gamma)
        G = lin.gn.dense(rows)
        lin.support_C += [G, -G]
        lin.support_l += [-tol - slide[:, axis], -tol + slide[:, axis]]

Functions:

Dexterous-hand grasps: an add-on of :class:~.retargeter.OTRetargeter (addons={"hand_grasp": {...}}).

The interaction terms copy the demonstrated hand-object distances. Where the demonstration is loose (fingers hovering 2-4 cm off a pole, the pole sunk into the palm) or the robot hand is of other proportions, copied distances do not make a grasp: the robot fingers hover, and a hand that starts or ends up on the wrong side of a thin part stays there. Three stages, each an option below (measurements in the recipe README):

• Grasp closure — after each frame's solve, the fingers of a grasping hand close on the object in a solve of their own (hand joints only).
• Palm cold start — frame 0 is also started with the touching palms pulled to the human palms' places; the solve that holds more objects on the human palms' side wins.
• Side re-approach — a palm that holds a thin part on the other side from the human palm's is brought round: the frame is solved again with that arm only and a palm goal, and the earlier frames are re-solved back to where the hand is clear.

Classes

HandGrasp

HandGrasp(rt, options)

Bases: Addon

Grasp closure, palm cold start and side re-approach of the rig's hands.

PARAMETER	DESCRIPTION
palms	{robot palm link: human wrist joint}, one entry per hand; or {link: [frame joint, joints...]} when the human palm's skin belongs to several joints (the SOMA rig's metacarpals). TYPE: dict[str, str | list[str]]
fingertips	{robot fingertip link: human distal-phalanx joint}; each fingertip belongs to the palm it hangs from. The finger's middle phalanx is the joint one less in the name's trailing digit (left_index3 -> left_index2); a name without one closes its distal part alone. TYPE: dict[str, str]
contact	(m) Human palm/finger distance that counts as touching. TYPE: float
closure	(m) While a hand grasps (its human palm, or two of its finger pads, within contact of an object), a finger whose human distal phalanx is within this of the object closes on it: its middle and distal phalanx probes whose human sources face the object are pulled onto the object, and so are the palm probes whose sources touch it. The closure is its own solve of the hand joints alone (arm, root and objects held), from the frame's solution; the next frame starts from the open-handed solution. 0 disables. TYPE: float
closure_weight	(m^-2 per link) Weight of the closure residual, normalised over the probes of each link. TYPE: float
cold_start_palm_weight	(m^-2) A second cold start: the posture fit with each palm whose human palm touches a free object at frame 0 also pulled to the human palm's place. Frame 0 is solved from both starts; the solve with more palms facing the object as the human palms do wins, then the one with fewer relaxed QPs, then the lower cost. 0 disables. TYPE: float
rest_palm_normal	The direction the human palms face in the body model's rest pose (SMPL-X: down, (0, 0, -1)); the robot's T-pose matches the rest pose, so its palm links face the same way there. TYPE: tuple[float, float, float]
side_reapproach	After each frame's solve, a palm whose human palm holds a thin part of a free object on the other side of the palm from the robot palm comes round (see _reapproach); info["side_reapproach"] is 1 at a corrected frame and a negative code at a dropped attempt (see _reapproach_palm). TYPE: bool

Source code in eden/extensions/retargeting/otretarget/grasp.py

def __init__(self, rt, options) -> None:
    super().__init__(rt, options)
    kin = rt.kin
    if not self.palms:
        raise ValueError(
            "HandGrasp: palms is empty; give {robot palm link: human wrist joint} per hand (the Wuji presets do)."
        )
    rt._refuse_unknown("hand_grasp palms keys", self.palms, kin.link_names, "the rig's links")
    rt._refuse_unknown("hand_grasp fingertips keys", self.fingertips, kin.link_names, "the rig's links")
    bad = [
        k for k in ("contact", "closure", "closure_weight", "cold_start_palm_weight") if not getattr(self, k) >= 0
    ]
    if bad:
        raise ValueError(f"HandGrasp: {', '.join(f'{k}={getattr(self, k)!r}' for k in bad)} must be >= 0.")
    if self.closure > 0 and not self.fingertips:
        raise ValueError("HandGrasp: closure needs `fingertips`; set them (the Wuji presets do) or leave it at 0.")
    if abs(float(np.linalg.norm(self.rest_palm_normal)) - 1.0) > 1e-6:
        raise ValueError(
            f"HandGrasp: rest_palm_normal {tuple(self.rest_palm_normal)!r} must be a unit vector (the direction the "
            "human palms face in the rest pose; SMPL-X: (0, 0, -1))."
        )
    palm_links = {kin.link_index(n) for n in self.palms}
    link_palm = np.full(len(kin.link_names), -1, dtype=np.int64)
    for li in range(len(kin.link_names)):
        cur = li
        while cur >= 0 and cur not in palm_links:
            cur = kin.link_parent[cur]
        link_palm[li] = cur
    self._link_palm = link_palm  # the palm link each link hangs from (-1: none)
    for tip in self.fingertips:
        if link_palm[kin.link_index(tip)] < 0:
            raise ValueError(
                f"HandGrasp: fingertip {tip!r} hangs from none of the palms {list(self.palms)}; add its hand's palm "
                f"link to `palms`."
            )
    # The hand joints (closure): every joint below a palm link, not the palm's own.
    hand = (link_palm[kin._joint_link] >= 0) & (link_palm[kin._joint_link] != kin._joint_link)
    self._hand_cols = np.zeros(kin.n_v, dtype=bool)
    self._hand_cols[kin._joint_var[hand]] = True
    self._normal = np.asarray(self.rest_palm_normal, dtype=np.float64)

Attributes

palms class-attribute instance-attribute

palms: dict[str, str | list[str]] = {}

fingertips class-attribute instance-attribute

fingertips: dict[str, str] = {}

contact class-attribute instance-attribute

contact: float = 0.01

closure class-attribute instance-attribute

closure: float = 0.0

closure_weight class-attribute instance-attribute

closure_weight: float = 2000.0

cold_start_palm_weight class-attribute instance-attribute

cold_start_palm_weight: float = 0.0

rest_palm_normal class-attribute instance-attribute

rest_palm_normal: tuple[float, float, float] = (0.0, 0.0, -1.0)

side_reapproach class-attribute instance-attribute

side_reapproach: bool = False

Methods:

setup

setup(human: HumanSurfaceMotion) -> None

Source code in eden/extensions/retargeting/otretarget/grasp.py

def setup(self, human: HumanSurfaceMotion) -> None:
    rt, kin = self.rt, self.rt.kin
    # Per palm link: the human joint whose frame is the palm's, and the joints whose skin is the palm.
    joints = {kin.link_index(n): [j] if isinstance(j, str) else list(j) for n, j in self.palms.items()}
    self._palm_joint = {li: human.joint_index(j[0]) for li, j in joints.items()}
    self._palm_parts = {li: [human.joint_index(n) for n in j] for li, j in joints.items()}
    # Fingers: (palm link, human distal joint) per fingertip; and per robot probe, the finger whose middle or
    # distal phalanx its human source lies on (closure).
    self._fingers = []
    self._closure_finger = np.full(len(rt.robot_probes), -1, dtype=np.int64)
    src_part = rt.human_probes.part[np.maximum(rt.src, 0)]
    for f, (tip, joint) in enumerate(self.fingertips.items()):
        j = human.joint_index(joint)
        if not (human.vertex_parts == j).any():
            raise ValueError(f"HandGrasp: fingertips[{tip!r}] = {joint!r} skins no vertex of this body model.")
        self._fingers.append((int(self._link_palm[kin.link_index(tip)]), j))
        if self.closure > 0:
            parts = [j]
            if joint[-1].isdigit():
                middle = joint[:-1] + str(int(joint[-1]) - 1)
                if middle not in human.joint_names:
                    raise ValueError(
                        f"HandGrasp: fingertips[{tip!r}] = {joint!r}: its middle phalanx {middle!r} (the name with "
                        f"its trailing digit less one) is not a joint of this body model."
                    )
                parts.append(human.joint_index(middle))
            self._closure_finger[(rt.src >= 0) & np.isin(src_part, parts)] = f
    self._side_state: dict = {}  # per palm link: the grasp being watched by ``side_reapproach``
    self._open_q = np.zeros((human.num_frames, kin.n_q))  # the open-hand solutions
    self._q_closed = None  # the previous frame's closed-hand configuration

cold_starts

cold_starts(human, obj_pos, obj_rot) -> list[ndarray]

Source code in eden/extensions/retargeting/otretarget/grasp.py

def cold_starts(self, human, obj_pos, obj_rot) -> list[np.ndarray]:
    if self.cold_start_palm_weight <= 0:
        return []
    palms = self._touching_palms(human, obj_pos, obj_rot)
    if not palms:
        return []
    return [
        self.rt._initial_configuration(
            human, obj_pos, obj_rot, pulls=palms, pull_weight=self.cold_start_palm_weight
        )
    ]

rank_start

rank_start(human, frame: Frame, q, obj_pos, obj_rot, n_starts: int) -> float

Minus the number of touching palms that hold the object on the human palm's side (with two starts or more).

Either start can be the wrong one: without the pull a hand of other reach started on the far side of a lamp pole, with it a palm pulled onto a chair's back rail put the arm through the chair. The cost alone does not see the side: the OMOMO palms sink into poles to their middle.

Source code in eden/extensions/retargeting/otretarget/grasp.py

def rank_start(self, human, frame: Frame, q, obj_pos, obj_rot, n_starts: int) -> float:
    """Minus the number of touching palms that hold the object on the human palm's side (with two starts or more).

    Either start can be the wrong one: without the pull a hand of other reach started on the far side of a lamp
    pole, with it a palm pulled onto a chair's back rail put the arm through the chair. The cost alone does not
    see the side: the OMOMO palms sink into poles to their middle.
    """
    if n_starts < 2:
        return 0.0
    palms = self._touching_palms(human, frame.demo_pos, frame.demo_rot)
    return -float(self._palms_facing(palms, human, frame.demo_pos, frame.demo_rot, q, obj_pos, obj_rot))

frame

frame(human: HumanSurfaceMotion, frame: Frame, forward: bool) -> None

Source code in eden/extensions/retargeting/otretarget/grasp.py

def frame(self, human: HumanSurfaceMotion, frame: Frame, forward: bool) -> None:
    rt = self.rt
    if self.closure > 0:
        for ci, oi in enumerate(rt._channel_object):
            if oi >= 0:
                self._grasp_closure(human, frame.t, frame.human_t[ci], frame.robot_t[ci], frame.demo_rot[oi])
    if forward and self._side_state:
        frame.extra["palm_goals"] = self._fading_goals(human, frame)

set_terms

set_terms(lin: Linearization, s: ProbeSet) -> None

Source code in eden/extensions/retargeting/otretarget/grasp.py

def set_terms(self, lin: Linearization, s: ProbeSet) -> None:
    targets = s.targets
    if s.kind != "robot" or targets.closed is None or lin.frame.stage != "closure":
        return
    rt = self.rt
    n_v = rt.kin.n_v
    d = s.d
    closing = targets.closed & np.isfinite(d)
    probe_palm = self._link_palm[rt.robot_probes.carrier]
    for pl in np.unique(probe_palm[closing]):
        # A hand closes once its palm faces the object: the palm probes whose human sources touch it are within
        # 1.5 cm of its nearest (its back is a palm's thickness, 3.3 cm, farther) and within 5 cm. Closed earlier,
        # the fist met the object first and kept it out of the palm.
        on_palm = rt.robot_probes.carrier == pl
        touch = closing & on_palm
        faces = touch.any() and d[touch].min() <= d[on_palm].min() + 0.015 and d[touch].min() < 0.05
        if pl < 0 or (touch.any() and not faces):
            closing &= probe_palm != pl
    kc = np.flatnonzero(closing)
    if not len(kc):
        return
    # Each closing probe onto the point of the object its human source is nearest (the side of a pole the human
    # thumb is on), through the hand's own joints only.
    _, inv, cnt = np.unique(rt.robot_probes.carrier[kc], return_inverse=True, return_counts=True)
    sw = np.sqrt(self.closure_weight / cnt[inv])
    miss = s.x[kc] - targets.witness[kc]
    for axis in range(3):
        gamma = np.zeros((len(kc), 3))
        gamma[:, axis] = 1.0
        rows = rt._rows("robot", s.ci, kc, s.world[kc], s.R, s.tau, lin.obj_pos, lin.link_pos, gamma)
        J = lin.gn.dense(rows)
        J[:, :n_v][:, ~self._hand_cols] = 0.0
        J[:, n_v:] = 0.0
        lin.gn.add_dense("grasp_closure", sw * miss[:, axis], sw[:, None] * J, 1.0)

terms

terms(lin: Linearization) -> None

Source code in eden/extensions/retargeting/otretarget/grasp.py

def terms(self, lin: Linearization) -> None:
    rt = self.rt
    n_x = rt.n_x
    link_pos, link_rot, link_jac = lin.link_pos, lin.link_rot, lin.link_jac
    for li, local, p_goal, R_goal, w in lin.frame.extra.get("palm_goals", ()):
        # The palm's probe centroid at the human palm's place relative to the object across the palm (the component
        # along the palm normal is left to the depth terms: the human palm sinks into what it holds, and a goal
        # there pressed the robot palm into a pole), and the palm at the human palm's orientation. The
        # non-penetration rows keep the hand on the side it starts on.
        n = R_goal @ rt._tpose_link_rot[li].T @ self._normal
        across = np.eye(3) - np.outer(n, n)
        p = link_pos[li] + link_rot[li] @ local
        J = np.vstack(
            [
                across @ (link_jac[li, 0:3] - skew(p - link_pos[li]) @ link_jac[li, 3:6]),
                _PALM_ROT_SCALE * link_jac[li, 3:6],
            ]
        )
        r = np.concatenate(
            [across @ (p - p_goal), _PALM_ROT_SCALE * axis_angle_from_quat(R_to_quat(link_rot[li] @ R_goal.T))]
        )
        lin.gn.add_dense("palm_goal", r, pad(J, n_x), w)

after_frame

after_frame(run: Run, frame: Frame, q, obj_pos, obj_rot, penetration: float)

Source code in eden/extensions/retargeting/otretarget/grasp.py

def after_frame(self, run: Run, frame: Frame, q, obj_pos, obj_rot, penetration: float):
    t = frame.t
    if self.side_reapproach:
        q, penetration = self._reapproach(run, frame, q, obj_pos, obj_rot, penetration)
    self._open_q[t] = q
    q_out = q
    if self.closure > 0 and self._fingers:
        q_out, closed_pen = self._close_hands(q, obj_pos, obj_rot, frame, self._q_closed)
        self._q_closed = q_out.copy()
        penetration = max(penetration, closed_pen)
    return q, q_out, penetration

Functions:

The demonstrator's surface: a skinned body mesh posed per frame, plus its joint frames.

OTRetarget reads interactions off the human surface, not its skeleton (Sec. III-A): probes are sampled on the posed mesh and each one is assigned to the body part that skins it. The skeleton's joint frames still drive the style cost (Sec. III-D). :class:HumanSurfaceMotion is that pair, in a z-up metric world, and is all :class:~eden.extensions.retargeting.otretarget.OTRetargeter needs from a body model — so SMPL-X (OMOMO, AMASS) and the SOMA rig (BONES-SEED BVH) feed it alike.

:class:~eden.datasets.utils.body_model.SkinnedBodyModel, Eden's linear-blend-skinning rig, poses both: :func:smplx_motion builds it from an SMPL-X model file, and :func:~eden.extensions.retargeting.otretarget.sources.load_soma_bvh from the SOMA rig.

Attributes

SMPLX_JOINT_NAMES module-attribute

SMPLX_JOINT_NAMES: tuple[str, ...] = ('pelvis', 'left_hip', 'right_hip', 'spine1', 'left_knee', 'right_knee', 'spine2', 'left_ankle', 'right_ankle', 'spine3', 'left_foot', 'right_foot', 'neck', 'left_collar', 'right_collar', 'head', 'left_shoulder', 'right_shoulder', 'left_elbow', 'right_elbow', 'left_wrist', 'right_wrist', 'jaw', 'left_eye', 'right_eye', *(f'left_{f}{i}' for f in ('index', 'middle', 'pinky', 'ring', 'thumb') for i in (1, 2, 3)), *(f'right_{f}{i}' for f in ('index', 'middle', 'pinky', 'ring', 'thumb') for i in (1, 2, 3)))

Y_UP_TO_Z_UP module-attribute

Y_UP_TO_Z_UP = np.array([[1.0, 0.0, 0.0], [0.0, 0.0, -1.0], [0.0, 1.0, 0.0]])

CANONICAL_TO_ROBOT module-attribute

CANONICAL_TO_ROBOT = np.array([[0.0, 0.0, 1.0], [1.0, 0.0, 0.0], [0.0, 1.0, 0.0]])

Classes

HumanSurfaceMotion dataclass

HumanSurfaceMotion(vertices: ndarray, faces: ndarray, skin_weights: ndarray, joint_names: list[str], joint_pos: ndarray, joint_rot: ndarray, rest_vertices: ndarray, rest_joint_pos: ndarray, rest_joint_rot: ndarray, fps: float)

A posed human surface over time, with its skeleton, in a z-up metric world.

ATTRIBUTE	DESCRIPTION
vertices	(T, V, 3) posed mesh vertices (world). TYPE: ndarray
faces	(F, 3) triangle indices. TYPE: ndarray
skin_weights	(V, J) skinning weights; a vertex belongs to the part of its heaviest joint. TYPE: ndarray
joint_names	The J skeleton joints. TYPE: list[str]
joint_pos	(T, J, 3) joint world positions. TYPE: ndarray
joint_rot	(T, J, 3, 3) joint world orientations. TYPE: ndarray
rest_vertices	(V, 3) the mesh in the model's rest (T-) pose, facing +x, z-up: the pose the transport plan is solved in and the style alignment offsets are measured at. TYPE: ndarray
rest_joint_pos	(J, 3) joint positions in that rest pose. TYPE: ndarray
rest_joint_rot	(J, 3, 3) joint orientations in that rest pose. TYPE: ndarray
fps	Frame rate. TYPE: float

Attributes

vertices instance-attribute

vertices: ndarray

faces instance-attribute

faces: ndarray

skin_weights instance-attribute

skin_weights: ndarray

joint_names instance-attribute

joint_names: list[str]

joint_pos instance-attribute

joint_pos: ndarray

joint_rot instance-attribute

joint_rot: ndarray

rest_vertices instance-attribute

rest_vertices: ndarray

rest_joint_pos instance-attribute

rest_joint_pos: ndarray

rest_joint_rot instance-attribute

rest_joint_rot: ndarray

fps instance-attribute

fps: float

vertex_parts class-attribute instance-attribute

vertex_parts: ndarray = field(init=False)

num_frames property

num_frames: int

Methods:

joint_index

joint_index(name: str) -> int

Index of joint name, raising with the available names if absent.

Source code in eden/extensions/retargeting/otretarget/human.py

def joint_index(self, name: str) -> int:
    """Index of joint ``name``, raising with the available names if absent."""
    if name not in self.joint_names:
        raise KeyError(f"Human joint {name!r} is not in this body model. Available: {list(self.joint_names)}.")
    return self.joint_names.index(name)

translated

translated(offset: ndarray) -> HumanSurfaceMotion

The same motion moved by a constant world offset (3,) (the rest pose is unchanged).

Source code in eden/extensions/retargeting/otretarget/human.py

def translated(self, offset: np.ndarray) -> HumanSurfaceMotion:
    """The same motion moved by a constant world offset ``(3,)`` (the rest pose is unchanged)."""
    offset = np.asarray(offset, dtype=np.float64)
    return HumanSurfaceMotion(
        vertices=(self.vertices + offset).astype(self.vertices.dtype),
        faces=self.faces,
        skin_weights=self.skin_weights,
        joint_names=list(self.joint_names),
        joint_pos=self.joint_pos + offset,
        joint_rot=self.joint_rot,
        rest_vertices=self.rest_vertices,
        rest_joint_pos=self.rest_joint_pos,
        rest_joint_rot=self.rest_joint_rot,
        fps=self.fps,
    )

floor_offset

floor_offset(foot_joints: tuple[str, ...], quantile: float = 0.5) -> float

Height of the floor the feet stand on, read off the motion itself.

The quantile of the per-frame lowest foot-vertex height. With at least one foot down in most frames (standing, walking, carrying, dancing), the median is the stance height: a capture whose body fit floats (or sinks) by a constant amount reports that amount, and the short flight phase of a jump does not move it. A low quantile would instead pick the frames where the fit penetrates.

PARAMETER	DESCRIPTION
foot_joints	Skeleton joints whose skinned vertices make up the feet. TYPE: tuple[str, ...]
quantile	Quantile over frames. TYPE: float DEFAULT: 0.5

Source code in eden/extensions/retargeting/otretarget/human.py

def floor_offset(self, foot_joints: tuple[str, ...], quantile: float = 0.5) -> float:
    """Height of the floor the feet stand on, read off the motion itself.

    The ``quantile`` of the per-frame lowest foot-vertex height. With at least one foot down in most
    frames (standing, walking, carrying, dancing), the median is the stance height: a capture whose
    body fit floats (or sinks) by a constant amount reports that amount, and the short flight phase of
    a jump does not move it. A low quantile would instead pick the frames where the fit penetrates.

    Parameters
    ----------
    foot_joints : tuple[str, ...]
        Skeleton joints whose skinned vertices make up the feet.
    quantile : float
        Quantile over frames.
    """
    feet = np.isin(self.vertex_parts, [self.joint_index(n) for n in foot_joints])
    return float(np.quantile(self.vertices[:, feet, 2].min(axis=1), quantile))

mirrored

mirrored(vertex_mirror: ndarray, joint_mirror: ndarray, local_mirror: ndarray, normal: ndarray, point: ndarray) -> HumanSurfaceMotion

The mirror image of this motion through a world plane: a left-handed demonstration made right-handed.

A reflection alone would turn the left hand into a right-shaped hand still labelled "left". The image is re-indexed so that every vertex and joint keeps its anatomical name: vertex k of the result is the reflection of vertex vertex_mirror[k] (its mirror partner on the body), joint j that of joint joint_mirror[j]. Joint frames become M R L (M the world reflection, L the reflection in the joint's own frame), proper rotations again; the rest pose is mirrored about its sagittal plane, which leaves a symmetric rest frame (and so every style alignment offset) unchanged. Face winding is reversed so normals still point out of the body.

PARAMETER	DESCRIPTION
vertex_mirror	(V,) permutation: the mirror partner of each vertex. TYPE: ndarray
joint_mirror	(J,) permutation: the mirror partner of each joint (left_* <-> right_*, the rest to themselves). TYPE: ndarray
local_mirror	(3, 3) the body model's sagittal reflection in a joint frame (SMPL-X: diag(-1, 1, 1)). TYPE: ndarray
normal	(3,) normal of the world mirror plane. TYPE: ndarray
point	(3,) a point on that plane. TYPE: ndarray

Source code in eden/extensions/retargeting/otretarget/human.py

def mirrored(
    self,
    vertex_mirror: np.ndarray,
    joint_mirror: np.ndarray,
    local_mirror: np.ndarray,
    normal: np.ndarray,
    point: np.ndarray,
) -> HumanSurfaceMotion:
    """The mirror image of this motion through a world plane: a left-handed demonstration made right-handed.

    A reflection alone would turn the left hand into a right-shaped hand still labelled "left". The image is
    re-indexed so that every vertex and joint keeps its anatomical name: vertex ``k`` of the result is the
    reflection of vertex ``vertex_mirror[k]`` (its mirror partner on the body), joint ``j`` that of joint
    ``joint_mirror[j]``. Joint frames become ``M R L`` (``M`` the world reflection, ``L`` the reflection in
    the joint's own frame), proper rotations again; the rest pose is mirrored about its sagittal plane,
    which leaves a symmetric rest frame (and so every style alignment offset) unchanged. Face winding is
    reversed so normals still point out of the body.

    Parameters
    ----------
    vertex_mirror : np.ndarray
        ``(V,)`` permutation: the mirror partner of each vertex.
    joint_mirror : np.ndarray
        ``(J,)`` permutation: the mirror partner of each joint (``left_*`` <-> ``right_*``, the rest to themselves).
    local_mirror : np.ndarray
        ``(3, 3)`` the body model's sagittal reflection in a joint frame (SMPL-X: ``diag(-1, 1, 1)``).
    normal : np.ndarray
        ``(3,)`` normal of the world mirror plane.
    point : np.ndarray
        ``(3,)`` a point on that plane.
    """
    n_verts, n_joints = len(self.rest_vertices), len(self.joint_names)
    vertex_mirror = np.asarray(vertex_mirror, dtype=np.int64)
    joint_mirror = np.asarray(joint_mirror, dtype=np.int64)
    for name, perm, size in (("vertex_mirror", vertex_mirror, n_verts), ("joint_mirror", joint_mirror, n_joints)):
        if perm.shape != (size,) or not np.array_equal(np.sort(perm), np.arange(size)):
            raise ValueError(f"HumanSurfaceMotion.mirrored: {name} must be a permutation of range({size}).")
    L = np.asarray(local_mirror, dtype=np.float64)
    if L.shape != (3, 3) or not np.isclose(np.linalg.det(L), -1.0) or not np.allclose(L @ L.T, np.eye(3)):
        raise ValueError(f"HumanSurfaceMotion.mirrored: local_mirror must be a 3x3 reflection; got {L.tolist()}.")
    n = np.asarray(normal, dtype=np.float64)
    n = n / np.linalg.norm(n)
    M = np.eye(3) - 2.0 * np.outer(n, n)
    shift = 2.0 * float(np.dot(np.asarray(point, dtype=np.float64), n)) * n

    def reflect(x: np.ndarray) -> np.ndarray:
        return x @ M + shift  # M is symmetric

    R_rest = np.diag([1.0, -1.0, 1.0])  # the rest pose faces +x with its lateral axis along y
    inverse = np.argsort(vertex_mirror)
    return HumanSurfaceMotion(
        vertices=reflect(self.vertices[:, vertex_mirror].astype(np.float64)).astype(self.vertices.dtype),
        faces=inverse[np.asarray(self.faces)][:, ::-1].copy(),
        skin_weights=self.skin_weights[vertex_mirror][:, joint_mirror],
        joint_names=list(self.joint_names),
        joint_pos=reflect(self.joint_pos[:, joint_mirror]),
        joint_rot=M @ self.joint_rot[:, joint_mirror] @ L,
        rest_vertices=self.rest_vertices[vertex_mirror] @ R_rest,
        rest_joint_pos=self.rest_joint_pos[joint_mirror] @ R_rest,
        rest_joint_rot=R_rest @ self.rest_joint_rot[joint_mirror] @ L,
        fps=self.fps,
    )

slice

slice(start: int = 0, stop: int | None = None, step: int = 1) -> HumanSurfaceMotion

The same motion restricted to frames start:stop:step (fps divided by step).

Source code in eden/extensions/retargeting/otretarget/human.py

def slice(self, start: int = 0, stop: int | None = None, step: int = 1) -> HumanSurfaceMotion:
    """The same motion restricted to frames ``start:stop:step`` (``fps`` divided by ``step``)."""
    sl = np.s_[start:stop:step]
    return HumanSurfaceMotion(
        vertices=self.vertices[sl],
        faces=self.faces,
        skin_weights=self.skin_weights,
        joint_names=list(self.joint_names),
        joint_pos=self.joint_pos[sl],
        joint_rot=self.joint_rot[sl],
        rest_vertices=self.rest_vertices,
        rest_joint_pos=self.rest_joint_pos,
        rest_joint_rot=self.rest_joint_rot,
        fps=self.fps / step,
    )

Functions:

smplx_motion

smplx_motion(model_path: str, poses: ndarray, trans: ndarray, betas: ndarray, fps: float, flat_hand_mean: bool = False, up_axis: str = 'y', v_template: ndarray | None = None) -> HumanSurfaceMotion

Pose SMPL-X from axis-angle parameters into a :class:HumanSurfaceMotion.

PARAMETER	DESCRIPTION
model_path	An SMPLX_<GENDER> model file (.npz or .pkl). TYPE: str
poses	(T, 156) [global_orient 3, body 63, left_hand 45, right_hand 45] (jaw and eyes are held at zero) or (T, 165) with [jaw 3, leye 3, reye 3] between body and hands. TYPE: ndarray
trans	(T, 3) SMPL-X translation. TYPE: ndarray
betas	Shape coefficients, up to the model's shape components: 300 for an SMPL-X v1.1 file (300 shape + 100 expression blend shapes), 10 for v1.0 (10 + 10). Coefficients past that must be zero. TYPE: ndarray
fps	Frame rate. TYPE: float
flat_hand_mean	False adds the model's mean hand pose to the hand parameters (the smplx package's default with use_pca=False); True reads them as offsets from a flat hand. TYPE: bool DEFAULT: False
up_axis	The world up axis of poses/trans; a y-up world is rotated to Eden's z-up. TYPE: ('y', 'z') DEFAULT: "y"
v_template	(10475, 3) a personal template mesh that replaces the model's mean shape (GRAB ships each subject as one, with betas = 0). betas are still added on top of it. TYPE: ndarray DEFAULT: None

RETURNS	DESCRIPTION
HumanSurfaceMotion

Source code in eden/extensions/retargeting/otretarget/human.py

def smplx_motion(
    model_path: str,
    poses: np.ndarray,
    trans: np.ndarray,
    betas: np.ndarray,
    fps: float,
    flat_hand_mean: bool = False,
    up_axis: str = "y",
    v_template: np.ndarray | None = None,
) -> HumanSurfaceMotion:
    """Pose SMPL-X from axis-angle parameters into a :class:`HumanSurfaceMotion`.

    Parameters
    ----------
    model_path : str
        An ``SMPLX_<GENDER>`` model file (``.npz`` or ``.pkl``).
    poses : np.ndarray
        ``(T, 156)`` ``[global_orient 3, body 63, left_hand 45, right_hand 45]`` (jaw and eyes are held
        at zero) or ``(T, 165)`` with ``[jaw 3, leye 3, reye 3]`` between body and hands.
    trans : np.ndarray
        ``(T, 3)`` SMPL-X translation.
    betas : np.ndarray
        Shape coefficients, up to the model's shape components: 300 for an SMPL-X v1.1 file (300 shape + 100
        expression blend shapes), 10 for v1.0 (10 + 10). Coefficients past that must be zero.
    fps : float
        Frame rate.
    flat_hand_mean : bool
        False adds the model's mean hand pose to the hand parameters (the ``smplx`` package's default
        with ``use_pca=False``); True reads them as offsets from a flat hand.
    up_axis : {"y", "z"}
        The world up axis of ``poses``/``trans``; a y-up world is rotated to Eden's z-up.
    v_template : np.ndarray, optional
        ``(10475, 3)`` a personal template mesh that replaces the model's mean shape (GRAB ships each
        subject as one, with ``betas`` = 0). ``betas`` are still added on top of it.

    Returns
    -------
    HumanSurfaceMotion
    """
    if up_axis not in ("y", "z"):
        raise ValueError(f"smplx_motion: up_axis={up_axis!r}; expected 'y' or 'z'.")
    data = body_model_arrays(str(model_path))
    poses = np.asarray(poses, dtype=np.float64)
    n_frames = len(poses)
    if poses.shape[1] == 156:
        full = np.concatenate([poses[:, :66], np.zeros((n_frames, 9)), poses[:, 66:]], axis=1)
    elif poses.shape[1] == 165:
        full = poses
    else:
        raise ValueError(
            f"smplx_motion: poses has {poses.shape[1]} columns; expected 156 "
            f"([global 3, body 63, lhand 45, rhand 45]) or 165 (with jaw/leye/reye before the hands)."
        )
    if not flat_hand_mean:
        full = full.copy()
        full[:, 75:120] += data["hands_meanl"]
        full[:, 120:165] += data["hands_meanr"]
    model = skinned_body_model(model_path, "smplx", betas, SMPLX_JOINT_NAMES, v_template)
    joints = model.bind_pos
    v_shaped = model.rest_vertices
    local_rot = quat_to_R(quat_from_axis_angle(full.reshape(n_frames, 55, 3)))
    # SMPL-X places the root joint at trans + J_0 (the root rotates about its joint, not the origin).
    root_pos = np.asarray(trans, dtype=np.float64) + joints[0]
    vertices, joint_pos, joint_rot = model.forward(local_rot, root_pos)
    if up_axis == "y":
        vertices = (vertices.astype(np.float64) @ Y_UP_TO_Z_UP.T).astype(np.float32)
        joint_pos = joint_pos @ Y_UP_TO_Z_UP.T
        joint_rot = Y_UP_TO_Z_UP[None, None] @ joint_rot
    rest_v, rest_p, rest_r = _rest_pose_motion(v_shaped, joints, model.bind_rot)
    return HumanSurfaceMotion(
        vertices=vertices,
        faces=model.faces,
        skin_weights=model.skin_weights,
        joint_names=list(SMPLX_JOINT_NAMES),
        joint_pos=joint_pos,
        joint_rot=joint_rot,
        rest_vertices=rest_v,
        rest_joint_pos=rest_p,
        rest_joint_rot=rest_r,
        fps=float(fps),
    )

Demonstration readers: dataset files to a :class:HumanSurfaceMotion plus :class:InteractionObject tracks.

• :func:load_grab_sequence — a raw GRAB capture (SMPL-X with a personal template, PCA hands, the object and the table), z-up, 120 fps.
• :func:mirror_demonstration — a left-handed SMPL-X demonstration made right-handed (body and objects).
• :func:load_omomo_sequence — OMOMO in its canonical SMPL-X form (the sequences_canonical/ layout of the eden_omomo release): human.npz (156-d SMPL-X poses with finger articulation, betas, trans, gender) and object.npz (axis-angle + translation of objects/<name>/<name>.obj), y-up.
• :func:load_soma_bvh — a SOMA-skeleton BVH (BONES-SEED) with the SOMA neutral body skinned onto it.

Attributes

SMPLX_FOOT_JOINTS module-attribute

SMPLX_FOOT_JOINTS = ('left_ankle', 'left_foot', 'right_ankle', 'right_foot')

SMPLX_HAND_JOINTS module-attribute

SMPLX_HAND_JOINTS = tuple(n for n in SMPLX_JOINT_NAMES if any(part in n for part in ('wrist', 'index', 'middle', 'ring', 'pinky', 'thumb')))

SMPLX_LOCAL_MIRROR module-attribute

SMPLX_LOCAL_MIRROR = np.diag([-1.0, 1.0, 1.0])

SOMA_FOOT_JOINTS module-attribute

SOMA_FOOT_JOINTS = ('LeftFoot', 'LeftToeBase', 'LeftToeEnd', 'RightFoot', 'RightToeBase', 'RightToeEnd')

Classes

Functions:

load_omomo_sequence

load_omomo_sequence(sequence_dir: str, objects_dir: str, smplx_dir: str, fps: float = 30.0, flat_hand_mean: bool = True, ground_human: bool = True) -> tuple[HumanSurfaceMotion, InteractionObject]

Read one canonical OMOMO sequence.

PARAMETER	DESCRIPTION
sequence_dir	.../sequences_canonical/<sub>_<object>_<clip>. TYPE: str
objects_dir	.../objects holding <name>/<name>.obj (metres). TYPE: str
smplx_dir	Directory with SMPLX_{NEUTRAL,MALE,FEMALE}.npz. TYPE: str
fps	The release's frame rate (30). TYPE: float DEFAULT: 30.0
flat_hand_mean	Hand pose convention. The release's hand poses are offsets from a flat hand: read with the mean hand added, the fingers of a box-carrying subject curl ~2 cm into the box. TYPE: bool DEFAULT: True
ground_human	Move the human vertically so its feet stand on z = 0 (:meth:HumanSurfaceMotion.floor_offset). Some subjects' body fits float above the floor the objects rest on (sub3: ~4.8 cm median over the clip, with the box at z = 0). OTRetarget works at native scale and treats the floor as an interaction channel, so it would faithfully copy that gap onto the robot's feet. The objects are not moved: they are already consistent with the floor. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
human	TYPE: HumanSurfaceMotion
obj	The object with its demonstrated trajectory, z-up. TYPE: InteractionObject

Source code in eden/extensions/retargeting/otretarget/sources.py

def load_omomo_sequence(
    sequence_dir: str,
    objects_dir: str,
    smplx_dir: str,
    fps: float = 30.0,
    flat_hand_mean: bool = True,
    ground_human: bool = True,
) -> tuple[HumanSurfaceMotion, InteractionObject]:
    """Read one canonical OMOMO sequence.

    Parameters
    ----------
    sequence_dir : str
        ``.../sequences_canonical/<sub>_<object>_<clip>``.
    objects_dir : str
        ``.../objects`` holding ``<name>/<name>.obj`` (metres).
    smplx_dir : str
        Directory with ``SMPLX_{NEUTRAL,MALE,FEMALE}.npz``.
    fps : float
        The release's frame rate (30).
    flat_hand_mean : bool
        Hand pose convention. The release's hand poses are offsets from a flat hand: read with the mean
        hand added, the fingers of a box-carrying subject curl ~2 cm into the box.
    ground_human : bool
        Move the human vertically so its feet stand on z = 0 (:meth:`HumanSurfaceMotion.floor_offset`).
        Some subjects' body fits float above the floor the objects rest on (sub3: ~4.8 cm median over the
        clip, with the box at z = 0). OTRetarget works at native scale and treats the floor as an
        interaction channel, so it would faithfully copy that gap onto the robot's feet. The objects are
        not moved: they are already consistent with the floor.

    Returns
    -------
    human : HumanSurfaceMotion
    obj : InteractionObject
        The object with its demonstrated trajectory, z-up.
    """
    human_npz = np.load(os.path.join(sequence_dir, "human.npz"), allow_pickle=True)
    object_npz = np.load(os.path.join(sequence_dir, "object.npz"), allow_pickle=True)
    gender = str(human_npz["gender"]).lower()
    model = os.path.join(smplx_dir, f"SMPLX_{gender.upper()}.npz")
    if not os.path.isfile(model):
        raise FileNotFoundError(
            f"load_omomo_sequence: {sequence_dir} is a {gender!r} subject but {model} does not exist; "
            f"point smplx_dir at the folder holding SMPLX_{{NEUTRAL,MALE,FEMALE}}.npz."
        )
    human = smplx_motion(
        model,
        human_npz["poses"],
        human_npz["trans"],
        human_npz["betas"],
        fps,
        flat_hand_mean=flat_hand_mean,
        up_axis="y",
    )
    if ground_human:
        human = human.translated(np.array([0.0, 0.0, -human.floor_offset(SMPLX_FOOT_JOINTS)]))
    name = str(object_npz["name"])
    mesh = trimesh.load(os.path.join(objects_dir, name, f"{name}.obj"), force="mesh", process=False)
    rot = Y_UP_TO_Z_UP[None] @ quat_to_R(quat_from_axis_angle(np.asarray(object_npz["angles"], dtype=np.float64)))
    pos = object_npz["trans"] @ Y_UP_TO_Z_UP.T
    poses = np.concatenate([pos, R_to_quat(rot)], axis=1)
    if len(poses) != human.num_frames:
        raise ValueError(
            f"load_omomo_sequence: {sequence_dir} has {human.num_frames} human frames but {len(poses)} object frames."
        )
    return human, InteractionObject(name=name, mesh=mesh, poses=poses)

load_grab_sequence

load_grab_sequence(sequence_path: str, grab_root: str, smplx_dir: str, start: int = 0, stop: int | None = None, step: int = 4, ground: bool = True, table_thickness: float = 0.03) -> tuple[HumanSurfaceMotion, list[InteractionObject]]

Read one raw GRAB capture (grab/<subject>/<object>_<intent>.npz) with its object and table.

The raw release is used, not a re-processed copy: its object poses fit the contact_meshes/*.ply scans as shipped (InterAct's re-centred objects/*.obj are off by up to 1.4 cm against the same poses), and it carries the table. Conventions, as in GRAB's own body and object models:

• z-up world, metres, 120 fps; step frames are skipped (4 -> 30 fps).
• Body: SMPL-X with the subject's personal template (vtemp) and betas = 0. fullpose already holds the 45-d hand poses expanded from the 24 PCA components with the mean hand added, so it is read as flat_hand_mean=True.
• Object and table: v_world = v_local @ R(global_orient) + transl, i.e. the column-vector rotation is R(global_orient).T.

The table mesh is a 5.5 mm slab. It is replaced by a table_thickness plate with the same footprint and the same top face (tilt included), held fixed at its first-frame pose: GRAB's table does not move.

PARAMETER	DESCRIPTION
sequence_path	.../grab/<subject>/<name>.npz. TYPE: str
grab_root	The folder the npz's mesh paths are relative to (holding tools/). TYPE: str
smplx_dir	Directory with SMPLX_{MALE,FEMALE}.npz. TYPE: str
start	Raw (120 fps) frame range and stride. TYPE: int DEFAULT: 0
stop	Raw (120 fps) frame range and stride. TYPE: int DEFAULT: 0
step	Raw (120 fps) frame range and stride. TYPE: int DEFAULT: 0
ground	Move the whole scene (body, object, table) vertically so the feet stand on z = 0. Body, object and table come from one marker capture, so they move together; moving only the body (as for OMOMO) would open a gap between the hand and the object. TYPE: bool DEFAULT: True
table_thickness	Thickness of the table plate (m). TYPE: float DEFAULT: 0.03

RETURNS	DESCRIPTION
human	TYPE: HumanSurfaceMotion
objects	The object, then the fixed table plate. TYPE: list[InteractionObject]

Source code in eden/extensions/retargeting/otretarget/sources.py

def load_grab_sequence(
    sequence_path: str,
    grab_root: str,
    smplx_dir: str,
    start: int = 0,
    stop: int | None = None,
    step: int = 4,
    ground: bool = True,
    table_thickness: float = 0.03,
) -> tuple[HumanSurfaceMotion, list[InteractionObject]]:
    """Read one raw GRAB capture (``grab/<subject>/<object>_<intent>.npz``) with its object and table.

    The raw release is used, not a re-processed copy: its object poses fit the ``contact_meshes/*.ply`` scans
    as shipped (InterAct's re-centred ``objects/*.obj`` are off by up to 1.4 cm against the same poses), and it
    carries the table. Conventions, as in GRAB's own body and object models:

    * z-up world, metres, 120 fps; ``step`` frames are skipped (4 -> 30 fps).
    * Body: SMPL-X with the subject's personal template (``vtemp``) and ``betas`` = 0. ``fullpose`` already
      holds the 45-d hand poses expanded from the 24 PCA components *with* the mean hand added, so it is read
      as ``flat_hand_mean=True``.
    * Object and table: ``v_world = v_local @ R(global_orient) + transl``, i.e. the column-vector rotation is
      ``R(global_orient).T``.

    The table mesh is a 5.5 mm slab. It is replaced by a ``table_thickness`` plate with the same footprint and
    the same top face (tilt included), held fixed at its first-frame pose: GRAB's table does not move.

    Parameters
    ----------
    sequence_path : str
        ``.../grab/<subject>/<name>.npz``.
    grab_root : str
        The folder the npz's mesh paths are relative to (holding ``tools/``).
    smplx_dir : str
        Directory with ``SMPLX_{MALE,FEMALE}.npz``.
    start, stop, step : int
        Raw (120 fps) frame range and stride.
    ground : bool
        Move the whole scene (body, object, table) vertically so the feet stand on z = 0. Body, object and
        table come from one marker capture, so they move together; moving only the body (as for OMOMO) would
        open a gap between the hand and the object.
    table_thickness : float
        Thickness of the table plate (m).

    Returns
    -------
    human : HumanSurfaceMotion
    objects : list[InteractionObject]
        The object, then the fixed ``table`` plate.
    """
    r = np.load(sequence_path, allow_pickle=True)
    n_raw = int(r["n_frames"])
    idx = np.arange(start, n_raw if stop is None else min(stop, n_raw), step)
    if len(idx) < 2:
        raise ValueError(
            f"load_grab_sequence: frames [{start}, {stop}) step {step} of {sequence_path} leave {len(idx)} frame(s); "
            f"the capture has {n_raw} raw frames."
        )
    gender = str(r["gender"]).lower()
    model = os.path.join(smplx_dir, f"SMPLX_{gender.upper()}.npz")
    if not os.path.isfile(model):
        raise FileNotFoundError(
            f"load_grab_sequence: {sequence_path} is a {gender!r} subject but {model} does not exist; "
            f"point smplx_dir at the folder holding SMPLX_{{MALE,FEMALE}}.npz."
        )
    body = r["body"].item()
    vtemp = np.asarray(trimesh.load(os.path.join(grab_root, body["vtemp"]), process=False).vertices)
    params = body["params"]
    human = smplx_motion(
        model,
        params["fullpose"][idx],
        params["transl"][idx],
        np.zeros(10),
        float(r["framerate"]) / step,
        flat_hand_mean=True,
        up_axis="z",
        v_template=vtemp,
    )

    def track(entry: dict) -> np.ndarray:
        aa = np.asarray(entry["params"]["global_orient"][idx], dtype=np.float64)
        rot = np.transpose(quat_to_R(quat_from_axis_angle(aa)), (0, 2, 1))
        return np.concatenate([entry["params"]["transl"][idx], R_to_quat(rot)], axis=1)

    obj_entry, table_entry = r["object"].item(), r["table"].item()
    mesh = trimesh.load(os.path.join(grab_root, obj_entry["object_mesh"]), force="mesh", process=False)
    obj = InteractionObject(name=str(r["obj_name"]), mesh=mesh, poses=track(obj_entry))

    table_pose = track(table_entry)
    drift = np.abs(table_pose[:, :3] - table_pose[0, :3]).max()
    if drift > 0.005:
        raise ValueError(
            f"load_grab_sequence: the table of {sequence_path} moves {drift * 100:.1f} cm over frames "
            f"[{start}, {idx[-1] + 1}); it is modelled as a fixed plate, so this capture is not supported."
        )
    slab = trimesh.load(os.path.join(grab_root, table_entry["table_mesh"]), force="mesh", process=False)
    R0 = quat_to_R(quat_from_axis_angle(np.asarray(table_entry["params"]["global_orient"][idx[0]], dtype=np.float64))).T
    thin = int(np.argmin(slab.extents))  # the slab's thickness axis, vertical in the world
    up = np.sign(R0[2, thin])
    lo, hi = slab.bounds
    top = hi[thin] if up > 0 else lo[thin]
    extents = slab.extents.copy()
    extents[thin] = table_thickness
    plate = trimesh.creation.box(extents=extents)
    centre = 0.5 * (lo + hi)
    centre[thin] = top - up * 0.5 * table_thickness
    plate.apply_translation(centre)
    table = InteractionObject(name="table", mesh=plate, poses=np.tile(table_pose[0], (len(idx), 1)), fixed=True)

    if ground:
        lift = np.array([0.0, 0.0, -human.floor_offset(SMPLX_FOOT_JOINTS)])
        human = human.translated(lift)
        for o in (obj, table):
            o.poses = o.poses.copy()
            o.poses[:, :3] += lift
    return human, [obj, table]

smplx_mirror_maps

smplx_mirror_maps(model_path: str, tolerance: float = 0.0001) -> tuple[ndarray, ndarray]

SMPL-X's left/right correspondences: the mirror partner of every vertex and every joint.

The model's mean template is symmetric about its sagittal plane (x = 0): each vertex reflected through it lands on another vertex (within 15 um), or on itself on the midline. The partners are found that way and checked to form an involution.

PARAMETER	DESCRIPTION
model_path	An SMPLX_<GENDER>.npz. TYPE: str
tolerance	Largest allowed gap (m) between a reflected vertex and its partner. TYPE: float DEFAULT: 0.0001

RETURNS	DESCRIPTION
vertex_mirror	(10475,) partner of each vertex. TYPE: ndarray
joint_mirror	(55,) partner of each joint (left_* <-> right_*). TYPE: ndarray

Source code in eden/extensions/retargeting/otretarget/sources.py

def smplx_mirror_maps(model_path: str, tolerance: float = 1e-4) -> tuple[np.ndarray, np.ndarray]:
    """SMPL-X's left/right correspondences: the mirror partner of every vertex and every joint.

    The model's mean template is symmetric about its sagittal plane (x = 0): each vertex reflected through it
    lands on another vertex (within 15 um), or on itself on the midline. The partners are found that way and
    checked to form an involution.

    Parameters
    ----------
    model_path : str
        An ``SMPLX_<GENDER>.npz``.
    tolerance : float
        Largest allowed gap (m) between a reflected vertex and its partner.

    Returns
    -------
    vertex_mirror : np.ndarray
        ``(10475,)`` partner of each vertex.
    joint_mirror : np.ndarray
        ``(55,)`` partner of each joint (``left_*`` <-> ``right_*``).
    """
    from scipy.spatial import cKDTree

    template = body_model_arrays(str(model_path))["v_template"].astype(np.float64)
    gap, vertex_mirror = cKDTree(template).query(template * np.array([-1.0, 1.0, 1.0]))
    if gap.max() > tolerance or not np.array_equal(vertex_mirror[vertex_mirror], np.arange(len(template))):
        raise ValueError(
            f"smplx_mirror_maps: {model_path}'s template is not mirror-symmetric about x = 0 (largest gap "
            f"{gap.max() * 1e3:.3f} mm, tolerance {tolerance * 1e3:.3f} mm); it is not an SMPL-X model."
        )
    names = list(SMPLX_JOINT_NAMES)
    swap = {"left_": "right_", "right_": "left_"}
    joint_mirror = np.array(
        [names.index(next((swap[p] + n[len(p) :] for p in swap if n.startswith(p)), n)) for n in names]
    )
    return vertex_mirror.astype(np.int64), joint_mirror

mirror_object

mirror_object(obj: InteractionObject, normal: ndarray, point: ndarray, local_mirror: ndarray = SMPLX_LOCAL_MIRROR) -> InteractionObject

The mirror image of an object track through a world plane (the object of a mirrored demonstration).

v_world = R v + p reflected is (M R L)(L v) + M p + s: the mesh is reflected in its own frame by L (face winding reversed, so it stays outward-facing) and the pose becomes M R L, a proper rotation. A chiral object becomes its mirror twin; a symmetric one comes out the same shape.

PARAMETER	DESCRIPTION
obj	The object track to mirror. TYPE: InteractionObject
normal	(3,) the world mirror plane. TYPE: ndarray
point	(3,) the world mirror plane. TYPE: ndarray
local_mirror	(3, 3) any reflection in the object frame. TYPE: ndarray DEFAULT: SMPLX_LOCAL_MIRROR

Source code in eden/extensions/retargeting/otretarget/sources.py

def mirror_object(
    obj: InteractionObject, normal: np.ndarray, point: np.ndarray, local_mirror: np.ndarray = SMPLX_LOCAL_MIRROR
) -> InteractionObject:
    """The mirror image of an object track through a world plane (the object of a mirrored demonstration).

    ``v_world = R v + p`` reflected is ``(M R L)(L v) + M p + s``: the mesh is reflected in its own frame by
    ``L`` (face winding reversed, so it stays outward-facing) and the pose becomes ``M R L``, a proper rotation.
    A chiral object becomes its mirror twin; a symmetric one comes out the same shape.

    Parameters
    ----------
    obj : InteractionObject
        The object track to mirror.
    normal, point : np.ndarray
        ``(3,)`` the world mirror plane.
    local_mirror : np.ndarray
        ``(3, 3)`` any reflection in the object frame.
    """
    n = np.asarray(normal, dtype=np.float64)
    n = n / np.linalg.norm(n)
    M = np.eye(3) - 2.0 * np.outer(n, n)
    shift = 2.0 * float(np.dot(np.asarray(point, dtype=np.float64), n)) * n
    L = np.asarray(local_mirror, dtype=np.float64)

    def reflect_mesh(mesh: trimesh.Trimesh | None) -> trimesh.Trimesh | None:
        if mesh is None:
            return None
        return trimesh.Trimesh(np.asarray(mesh.vertices) @ L.T, np.asarray(mesh.faces)[:, ::-1], process=False)

    rot = M[None] @ quat_to_R(obj.poses[:, 3:]) @ L
    pos = obj.poses[:, :3] @ M + shift
    return InteractionObject(
        name=obj.name,
        mesh=reflect_mesh(obj.mesh),
        poses=np.concatenate([pos, R_to_quat(rot)], axis=1),
        demo_mesh=reflect_mesh(obj.demo_mesh),
        fixed=obj.fixed,
        weight=obj.weight,
    )

mirror_demonstration

mirror_demonstration(human: HumanSurfaceMotion, objects: list[InteractionObject], smplx_model_path: str) -> tuple[HumanSurfaceMotion, list[InteractionObject]]

Mirror an SMPL-X demonstration and its objects through the demonstrator's first-frame sagittal plane.

The plane holds the pelvis and the pelvis' forward and up directions at frame 0, so the mirrored person starts where the original did, facing the same way, with every left/right role swapped.

Source code in eden/extensions/retargeting/otretarget/sources.py

def mirror_demonstration(
    human: HumanSurfaceMotion, objects: list[InteractionObject], smplx_model_path: str
) -> tuple[HumanSurfaceMotion, list[InteractionObject]]:
    """Mirror an SMPL-X demonstration and its objects through the demonstrator's first-frame sagittal plane.

    The plane holds the pelvis and the pelvis' forward and up directions at frame 0, so the mirrored person
    starts where the original did, facing the same way, with every left/right role swapped.
    """
    vertex_mirror, joint_mirror = smplx_mirror_maps(smplx_model_path)
    pelvis = human.joint_index("pelvis")
    lateral = human.joint_rot[0, pelvis][:, 0].copy()  # SMPL-X model x: the body's left
    lateral[2] = 0.0
    if np.linalg.norm(lateral) < 1e-6:
        raise ValueError("mirror_demonstration: the pelvis' lateral axis is vertical at frame 0; no sagittal plane.")
    point = human.joint_pos[0, pelvis]
    mirrored = human.mirrored(vertex_mirror, joint_mirror, SMPLX_LOCAL_MIRROR, lateral, point)
    return mirrored, [mirror_object(o, lateral, point) for o in objects]

load_soma_bvh

load_soma_bvh(bvh_path: str, soma_model_path: str, fps: float = 30.0, ground_human: bool = True) -> HumanSurfaceMotion

Read a SOMA-skeleton BVH (BONES-SEED) and skin the SOMA neutral body onto it.

The BVH drives the SOMA rig's 78 joints directly (same joint names and joint frames); the neutral mesh of SOMA_neutral.npz is scaled to the actor's overall bone length and skinned to the BVH's world joint transforms, so every limb, foot included, follows the captured skeleton.

PARAMETER	DESCRIPTION
bvh_path	A SOMA-rig .bvh (y-up, centimetres). TYPE: str
soma_model_path	SOMA_neutral.npz (from the nvidia/soma-x release). TYPE: str
fps	Output frame rate; the capture rate must be an integer multiple of it. TYPE: float DEFAULT: 30.0
ground_human	Move the motion vertically so the feet stand on z = 0 (:meth:HumanSurfaceMotion.floor_offset). TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
HumanSurfaceMotion

Source code in eden/extensions/retargeting/otretarget/sources.py

def load_soma_bvh(
    bvh_path: str,
    soma_model_path: str,
    fps: float = 30.0,
    ground_human: bool = True,
) -> HumanSurfaceMotion:
    """Read a SOMA-skeleton BVH (BONES-SEED) and skin the SOMA neutral body onto it.

    The BVH drives the SOMA rig's 78 joints directly (same joint names and joint frames); the neutral
    mesh of ``SOMA_neutral.npz`` is scaled to the actor's overall bone length and skinned to the BVH's
    world joint transforms, so every limb, foot included, follows the captured skeleton.

    Parameters
    ----------
    bvh_path : str
        A SOMA-rig ``.bvh`` (y-up, centimetres).
    soma_model_path : str
        ``SOMA_neutral.npz`` (from the ``nvidia/soma-x`` release).
    fps : float
        Output frame rate; the capture rate must be an integer multiple of it.
    ground_human : bool
        Move the motion vertically so the feet stand on z = 0 (:meth:`HumanSurfaceMotion.floor_offset`).

    Returns
    -------
    HumanSurfaceMotion
    """
    from scipy.sparse import csc_matrix

    from eden.datasets.utils.body_model import SkinnedBodyModel
    from eden.datasets.utils.bvh import forward_kinematics_matrices, parse_bvh
    from eden.extensions.retargeting.otretarget.human import CANONICAL_TO_ROBOT

    model = np.load(soma_model_path, allow_pickle=True)
    names = [str(n) for n in model["joint_names"]]
    parents = model["joint_parent_ids"].astype(np.int64).copy()
    parents[0] = -1
    bvh = parse_bvh(bvh_path)
    missing = [n for n in names if n not in bvh.joint_names]
    if missing:
        raise ValueError(
            f"load_soma_bvh: {bvh_path} lacks SOMA joints {missing}; it is not a SOMA-rig BVH "
            f"(its joints: {bvh.joint_names[:10]}...)."
        )
    step = bvh.framerate / fps
    if abs(step - round(step)) > 1e-3 or round(step) < 1:
        raise ValueError(
            f"load_soma_bvh: the capture runs at {bvh.framerate:.3f} fps, not an integer multiple of fps={fps}; "
            f"pick an fps that divides it (e.g. {bvh.framerate / 4:g})."
        )
    order = [bvh.joint_names.index(n) for n in names]
    pos, rot = forward_kinematics_matrices(bvh)
    pos, rot = pos[:, order], rot[:, order]
    rot, pos = rot[:: int(round(step))], pos[:: int(round(step))]

    bind = model["bind_pose_world"].astype(np.float64)
    bind_rot, bind_pos = bind[:, :3, :3], bind[:, :3, 3]
    # Scale the neutral body to the actor: ratio of total bone length, BVH offsets vs. SOMA bind.
    bvh_len = sum(np.linalg.norm(bvh.joints[bvh.joint_names.index(n)].offset) for n in names[2:])
    soma_len = sum(np.linalg.norm(bind_pos[j] - bind_pos[parents[j]]) for j in range(2, len(names)))
    scale = bvh_len / soma_len
    weights = csc_matrix(
        (model["skinning_weights_data"], model["skinning_weights_indices"], model["skinning_weights_indptr"]),
        shape=tuple(model["skinning_weights_shape"]),
    ).toarray()
    weights /= np.maximum(weights.sum(axis=1, keepdims=True), 1e-12)
    cm = 0.01
    body = SkinnedBodyModel(
        rest_vertices=model["mean"].astype(np.float64) * scale * cm,
        faces=model["triangles"].astype(np.int64),
        skin_weights=weights,
        parents=parents,
        bind_rot=bind_rot,
        bind_pos=bind_pos * scale * cm,
        joint_names=names,
    )
    vertices = body.skin(rot, pos * cm)
    joint_pos = pos * cm @ Y_UP_TO_Z_UP.T
    joint_rot = Y_UP_TO_Z_UP[None, None] @ rot
    vertices = (vertices.astype(np.float64) @ Y_UP_TO_Z_UP.T).astype(np.float32)

    # Rest pose for the transport plan: the rig's own T-pose (its bind pose is an A-pose).
    tpose = model["t_pose_world"].astype(np.float64)
    t_rot, t_pos = tpose[:, :3, :3], tpose[:, :3, 3] * scale * cm
    rest_v = body.skin(t_rot[None], t_pos[None])[0].astype(np.float64) @ CANONICAL_TO_ROBOT.T
    rest_p = t_pos @ CANONICAL_TO_ROBOT.T
    lift = -rest_v[:, 2].min()
    human = HumanSurfaceMotion(
        vertices=vertices,
        faces=body.faces,
        skin_weights=weights,
        joint_names=names,
        joint_pos=joint_pos,
        joint_rot=joint_rot,
        rest_vertices=rest_v + np.array([0.0, 0.0, lift]),
        rest_joint_pos=rest_p + np.array([0.0, 0.0, lift]),
        rest_joint_rot=CANONICAL_TO_ROBOT[None] @ t_rot,
        fps=float(fps),
    )
    if ground_human:
        human = human.translated(np.array([0.0, 0.0, -human.floor_offset(SOMA_FOOT_JOINTS)]))
    return human

Untracked supports: the tables and stools a demonstrated object rests on but the capture does not track.

A demonstration such as "put the monitor on the table" tracks the monitor, not the table. Retargeted without the table, nothing holds the resting object up: within the interaction range the robot's hands pull it down toward them (up to 22 cm on HiPHI). :func:find_untracked_supports reads each support off the object track and gives it back as a thin plate, which the retargeter takes as a fixed object.

A support is found where the object rests still above the floor with no hand on it (so it is not being held still in the air), and only if it could be resting on a level surface: one horizontal level of the object (its lowest one, or the seat of an upside-down chair whose backrest hangs over the table edge) must hold the object's centre of mass in plan view, with nothing of the object directly below it. A tilted object touching down on one corner, or one floating a few centimetres over the floor (a capture offset, not furniture), gives no support.

The plate is the support's top only: a block or a slab on legs fills the space where the demonstrator's (and so the robot's) legs stand against the table, and the non-penetration rows then reject every robot step.

Classes

Support dataclass

Support(name: str, height: float, center: ndarray, yaw: float, extents: ndarray, frames: tuple[int, int])

A plate standing in for an untracked support.

ATTRIBUTE	DESCRIPTION
name	The fixed object's name (support<i>). TYPE: str
height	Height of the plate's top surface (m): the level the object rests on. TYPE: float
center	(2,) horizontal position of the plate's centre. TYPE: ndarray
yaw	Plate heading (rad) about the vertical. TYPE: float
extents	(3,) plate size along its own axes (m); the last is its thickness. TYPE: ndarray
frames	The resting segment [start, stop) the support was read from. TYPE: tuple[int, int]

Attributes

name instance-attribute

name: str

height instance-attribute

height: float

center instance-attribute

center: ndarray

yaw instance-attribute

yaw: float

extents instance-attribute

extents: ndarray

frames instance-attribute

frames: tuple[int, int]

Methods:

pose

pose() -> ndarray

(7,) [pos, quat_wxyz] of the plate's centre.

Source code in eden/extensions/retargeting/otretarget/supports.py

def pose(self) -> np.ndarray:
    """``(7,)`` ``[pos, quat_wxyz]`` of the plate's centre."""
    z = self.height - 0.5 * self.extents[2]
    return np.r_[self.center[0], self.center[1], z, quat_from_axis_angle(np.array([0.0, 0.0, self.yaw]))]

to_dict

to_dict() -> dict

JSON-ready record: name, top height, plate pose [pos, quat_wxyz], extents and source frames.

Source code in eden/extensions/retargeting/otretarget/supports.py

def to_dict(self) -> dict:
    """JSON-ready record: name, top height, plate pose ``[pos, quat_wxyz]``, extents and source frames."""
    return {
        "name": self.name,
        "height": float(self.height),
        "pose_wxyz": self.pose().tolist(),
        "extents": [float(e) for e in self.extents],
        "frames": [int(f) for f in self.frames],
    }

as_object

as_object(n_frames: int) -> InteractionObject

The plate as a fixed :class:InteractionObject over n_frames frames.

Source code in eden/extensions/retargeting/otretarget/supports.py

def as_object(self, n_frames: int) -> InteractionObject:
    """The plate as a fixed :class:`InteractionObject` over ``n_frames`` frames."""
    mesh = trimesh.creation.box(extents=self.extents)
    return InteractionObject(self.name, mesh, np.tile(self.pose(), (n_frames, 1)), fixed=True)

Functions:

find_untracked_supports

find_untracked_supports(obj: InteractionObject, hand_points: ndarray, *, min_height: float = 0.15, floor_gap: float = 0.05, static_speed: float = 0.002, static_turn_deg: float = 0.5, hand_clearance: float = 0.06, min_frames: int = 5, level_band: float = 0.015, horizontal_deg: float = 18.0, com_tolerance: float = 0.01, margin: float = 0.02, clearance: float = 0.01, min_side: float = 0.1, thickness: float = 0.03) -> list[Support]

The untracked supports obj rests on in a demonstration (see the module docstring).

PARAMETER	DESCRIPTION
obj	The tracked object, z-up, floor at z = 0. TYPE: InteractionObject
hand_points	(T, H, 3) world points on the demonstrator's hands, same frames as obj.poses. TYPE: ndarray
min_height	A resting level lower than this is the floor seen through a capture offset, not furniture. TYPE: float DEFAULT: 0.15
floor_gap	The object's lowest point must be this far above the floor for a frame to count as resting elevated. TYPE: float DEFAULT: 0.05
static_speed	Per-frame translation (m) and rotation (deg) below which the object is still. TYPE: float DEFAULT: 0.002
static_turn_deg	Per-frame translation (m) and rotation (deg) below which the object is still. TYPE: float DEFAULT: 0.002
hand_clearance	Every hand point must be this far from the object (m): a held object is not resting. TYPE: float DEFAULT: 0.06
min_frames	Shortest resting segment. TYPE: int DEFAULT: 5
level_band	Thickness of one horizontal level of the object (m). TYPE: float DEFAULT: 0.015
horizontal_deg	Faces within this angle of horizontal (either facing) make up the levels. TYPE: float DEFAULT: 18.0
com_tolerance	How far outside the level's footprint the centre of mass may lie (m). TYPE: float DEFAULT: 0.01
margin	The plate reaches this far beyond the level's footprint (m), before clearing hanging parts. TYPE: float DEFAULT: 0.02
clearance	Gap kept between the plate and any part of the object below the level (m). TYPE: float DEFAULT: 0.01
min_side	A plate narrower than this (after clearing hanging parts) is no support. TYPE: float DEFAULT: 0.1
thickness	Plate thickness (m). TYPE: float DEFAULT: 0.03

RETURNS	DESCRIPTION
list[Support]	One per distinct support (resting segments at the same place and height share one).

Source code in eden/extensions/retargeting/otretarget/supports.py

def find_untracked_supports(
    obj: InteractionObject,
    hand_points: np.ndarray,
    *,
    min_height: float = 0.15,
    floor_gap: float = 0.05,
    static_speed: float = 0.002,
    static_turn_deg: float = 0.5,
    hand_clearance: float = 0.06,
    min_frames: int = 5,
    level_band: float = 0.015,
    horizontal_deg: float = 18.0,
    com_tolerance: float = 0.01,
    margin: float = 0.02,
    clearance: float = 0.01,
    min_side: float = 0.1,
    thickness: float = 0.03,
) -> list[Support]:
    """The untracked supports ``obj`` rests on in a demonstration (see the module docstring).

    Parameters
    ----------
    obj : InteractionObject
        The tracked object, z-up, floor at ``z = 0``.
    hand_points : np.ndarray
        ``(T, H, 3)`` world points on the demonstrator's hands, same frames as ``obj.poses``.
    min_height : float
        A resting level lower than this is the floor seen through a capture offset, not furniture.
    floor_gap : float
        The object's lowest point must be this far above the floor for a frame to count as resting elevated.
    static_speed, static_turn_deg : float
        Per-frame translation (m) and rotation (deg) below which the object is still.
    hand_clearance : float
        Every hand point must be this far from the object (m): a held object is not resting.
    min_frames : int
        Shortest resting segment.
    level_band : float
        Thickness of one horizontal level of the object (m).
    horizontal_deg : float
        Faces within this angle of horizontal (either facing) make up the levels.
    com_tolerance : float
        How far outside the level's footprint the centre of mass may lie (m).
    margin : float
        The plate reaches this far beyond the level's footprint (m), before clearing hanging parts.
    clearance : float
        Gap kept between the plate and any part of the object below the level (m).
    min_side : float
        A plate narrower than this (after clearing hanging parts) is no support.
    thickness : float
        Plate thickness (m).

    Returns
    -------
    list[Support]
        One per distinct support (resting segments at the same place and height share one).
    """
    poses = np.asarray(obj.poses, dtype=np.float64)
    hand_points = np.asarray(hand_points, dtype=np.float64)
    if hand_points.ndim != 3 or hand_points.shape[0] != len(poses) or hand_points.shape[2] != 3:
        raise ValueError(
            f"find_untracked_supports: hand_points has shape {hand_points.shape}; expected (T, H, 3) with "
            f"T = {len(poses)}, the object's frame count."
        )
    mesh = obj.mesh
    verts = np.asarray(mesh.vertices, dtype=np.float64)
    faces = np.asarray(mesh.faces, dtype=np.int64)
    com = _center_of_mass(mesh)
    rot = quat_to_R(poses[:, 3:])
    pos = poses[:, :3]
    zmin = np.einsum("tab,vb->tva", rot, verts)[:, :, 2].min(axis=1) + pos[:, 2]
    aabb = igl.AABB()
    aabb.init(verts, faces)
    # Dense surface points for the parts hanging below a level: a coarse mesh's vertices miss a face crossing the plate.
    n_samples = int(min(max(mesh.area / 0.005**2, 1000), 50000))
    surface = np.vstack([verts, trimesh.sample.sample_surface_even(mesh, n_samples, seed=0)[0]])
    speed = np.r_[0.0, np.linalg.norm(np.diff(pos, axis=0), axis=1)]
    step = axis_angle_from_quat(R_to_quat(rot[1:] @ rot[:-1].transpose(0, 2, 1)))
    turn = np.r_[0.0, np.degrees(np.linalg.norm(step, axis=1))]
    resting = (zmin > floor_gap) & (speed < static_speed) & (turn < static_turn_deg)
    for t in np.flatnonzero(resting):  # the hand test only where the rest already holds: it is the costly one
        local = np.ascontiguousarray((hand_points[t] - pos[t]) @ rot[t])
        resting[t] = aabb.squared_distance(verts, faces, local)[0].min() > hand_clearance**2

    supports: list[Support] = []
    for start, stop in _runs(resting, min_frames):
        k = (start + stop) // 2
        found = _resting_level(
            verts @ rot[k].T + pos[k],
            faces,
            surface @ rot[k].T + pos[k],
            com @ rot[k].T + pos[k],
            min_height,
            level_band,
            np.cos(np.radians(horizontal_deg)),
            com_tolerance,
            margin,
            clearance,
            min_side,
        )
        if found is None:
            continue
        height, center, yaw, size = found
        if any(abs(s.height - height) < 0.02 and np.linalg.norm(s.center - center) < 0.1 for s in supports):
            continue
        extents = np.array([size[0], size[1], thickness])
        supports.append(Support(f"support{len(supports)}", height, center, yaw, extents, (start, stop)))
    return supports

The retargeted robot as one kinematic tree over several Eden entities, with analytic Jacobians.

A dexterous humanoid is usually a composite (G1Wuji, OliWuji): a floating-base body entity plus hand entities welded to its wrists, each with its own qpos. The solver needs one configuration and one tangent space across all of them, and the Jacobian of any point on any link with respect to every degree of freedom that moves it — including the body's joints for a point on a fingertip, which no single entity's get_jacobian provides.

:class:RigKinematics builds that from the entities' joint anchors and axes. Forward kinematics stays Genesis' (set_qpos + link poses), so the link frames, joint placements and the meshes probes are sampled from all come from one source. The tangent space is::

v = [base translation (3, world), base rotation (3, world rotation vector), joints...]

for a floating root (the base block is absent for a fixed one), with the retraction p <- p + dp and R <- exp(dtheta) R — the same convention the paper uses for object poses, and independent of how Genesis parameterises its free joint internally.

Classes

RigKinematics

RigKinematics(env: EnvBase, entity_names: list[str])

Kinematics of a robot spread over a root entity and welded child entities.

PARAMETER	DESCRIPTION
env	A built env holding the entities (num_envs of 1 is enough; env 0 is used). TYPE: EnvBase
entity_names	The root entity first (floating or fixed base), then every entity welded below it. Each child must be attached (no free root of its own) and every joint must be fixed, revolute or prismatic, apart from the root's floating joint. TYPE: list[str]

ATTRIBUTE	DESCRIPTION
link_names	Every link of the rig, root entity first; names must be unique across the entities. TYPE: list[str]
n_v	Tangent dimension (6 for a floating base, plus one per joint). TYPE: int
n_q	Configuration dimension (7 for a floating base, plus one per joint). TYPE: int
var_names	One name per tangent entry (base_x ... base_rz, then the joint names). TYPE: list[str]

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def __init__(self, env: EnvBase, entity_names: list[str]) -> None:
    missing = [n for n in entity_names if n not in env.entities]
    if missing:
        raise ValueError(
            f"RigKinematics: entities {missing} are not in this env. Entities: {sorted(env.entities)}."
        )
    self.entities = [env.entities[n] for n in entity_names]
    self.entity_names = list(entity_names)
    root = self.entities[0]
    self.floating = not root.has_immobile_base
    for name, ent in zip(entity_names[1:], self.entities[1:]):
        if not ent.has_immobile_base:
            raise ValueError(
                f"RigKinematics: entity {name!r} has a free root, so it is not welded to {entity_names[0]!r}. "
                f"List only the root and the entities attached below it."
            )
    self.device = root.device

    # -- links ---------------------------------------------------------
    self.link_names: list[str] = []
    self._link_global: list[int] = []
    self._entity_link_slices: list[slice] = []
    for ent in self.entities:
        start = len(self.link_names)
        for link in ent.links:
            self.link_names.append(link.name)
            self._link_global.append(link.idx)
        self._entity_link_slices.append(slice(start, len(self.link_names)))
    dupes = sorted({n for n in self.link_names if self.link_names.count(n) > 1})
    if dupes:
        raise ValueError(f"RigKinematics: link names {dupes} appear in more than one entity of {entity_names}.")
    global_to_local = {g: i for i, g in enumerate(self._link_global)}
    all_links = {link.idx: link for ent in self.entities for link in ent.links}
    self.link_parent = np.array(
        [global_to_local.get(all_links[g].parent_idx, -1) for g in self._link_global], dtype=np.int64
    )
    roots = [self.link_names[i] for i in np.flatnonzero(self.link_parent < 0)]
    if len(roots) != 1:
        raise ValueError(
            f"RigKinematics: entities {entity_names} form {len(roots)} trees (roots {roots}); weld every child "
            f"entity below the root entity so the rig is a single tree."
        )

    # -- variables -----------------------------------------------------
    self.var_names: list[str] = (
        ["base_x", "base_y", "base_z", "base_rx", "base_ry", "base_rz"] if self.floating else []
    )
    self._entity_var_slices: list[slice] = []
    self._entity_q_slices: list[slice] = []
    q_cursor = 7 if self.floating else 0
    for ent in self.entities:
        start = len(self.var_names)
        self.var_names.extend(ent.dofs_name)
        self._entity_var_slices.append(slice(start, len(self.var_names)))
        n_joint_q = len(ent.dofs_name)
        own_q = slice(q_cursor - (7 if (ent is root and self.floating) else 0), q_cursor + n_joint_q)
        self._entity_q_slices.append(own_q)
        q_cursor += n_joint_q
        expect_q = n_joint_q + (7 if (ent is root and self.floating) else 0)
        if ent.n_qs != expect_q or ent.n_dofs != n_joint_q + (6 if (ent is root and self.floating) else 0):
            raise ValueError(
                f"RigKinematics: entity has n_qs={ent.n_qs}, n_dofs={ent.n_dofs} for {n_joint_q} named DOFs; only "
                f"one-DOF (revolute/prismatic) joints are supported besides the root's floating joint."
            )
    self.n_v = len(self.var_names)
    self.n_q = q_cursor
    self.n_base_v = 6 if self.floating else 0
    self.n_base_q = 7 if self.floating else 0

    # -- joints: child link, type, anchor/axis in the child link frame ----
    self._set_neutral()
    link_pos, link_rot = self.link_poses()
    joint_link, joint_var, joint_type, joint_anchor, joint_axis = [], [], [], [], []
    for e_idx, ent in enumerate(self.entities):
        names = list(ent.dofs_name)
        for joint in ent._entity.joints:
            jtype = int(joint.type)
            if jtype in (int(gs.JOINT_TYPE.FIXED), int(gs.JOINT_TYPE.FREE)):
                continue
            if jtype not in (int(gs.JOINT_TYPE.REVOLUTE), int(gs.JOINT_TYPE.PRISMATIC)):
                raise ValueError(
                    f"RigKinematics: joint {joint.name!r} of entity {self.entity_names[e_idx]!r} has type "
                    f"{gs.JOINT_TYPE(jtype).name}; only revolute and prismatic joints can be retargeted."
                )
            if joint.name not in names:
                raise ValueError(
                    f"RigKinematics: joint {joint.name!r} is not in entity {self.entity_names[e_idx]!r}'s dofs_name "
                    f"{names}; every actuated joint must be a named DOF."
                )
            li = global_to_local[joint.link.idx]
            anchor = joint.get_anchor_pos().reshape(-1, 3)[0].cpu().numpy().astype(np.float64)
            axis = joint.get_anchor_axis().reshape(-1, 3)[0].cpu().numpy().astype(np.float64)
            joint_link.append(li)
            joint_var.append(self._entity_var_slices[e_idx].start + names.index(joint.name))
            joint_type.append(jtype)
            joint_anchor.append(link_rot[li].T @ (anchor - link_pos[li]))
            joint_axis.append(link_rot[li].T @ axis)
    self._joint_link = np.array(joint_link, dtype=np.int64)
    self._joint_var = np.array(joint_var, dtype=np.int64)
    self._joint_revolute = np.array(joint_type) == int(gs.JOINT_TYPE.REVOLUTE)
    self._joint_anchor_local = np.array(joint_anchor).reshape(-1, 3)
    self._joint_axis_local = np.array(joint_axis).reshape(-1, 3)

    # joint_moves_link[l, k]: joint k moves link l (k's child link is l or one of l's ancestors).
    n_links = len(self.link_names)
    chain = [set() for _ in range(n_links)]
    for li in range(n_links):
        cur = li
        while cur >= 0:
            chain[li].add(cur)
            cur = self.link_parent[cur]
    self.joint_moves_link = np.array(
        [[self._joint_link[k] in chain[li] for k in range(len(self._joint_link))] for li in range(n_links)],
        dtype=bool,
    ).reshape(n_links, len(self._joint_link))

    # -- limits (joint DOFs only; dofs_name order) ------------------------
    lower, upper = [], []
    for ent in self.entities:
        lo, hi = (
            b.reshape(-1, len(ent.dofs_name))[0].cpu().numpy().astype(np.float64) for b in ent.get_dofs_limit()
        )
        lower.append(lo)
        upper.append(hi)
    self.joint_lower = np.concatenate(lower)
    self.joint_upper = np.concatenate(upper)

Attributes

entities instance-attribute

entities = [env.entities[n] for n in entity_names]

entity_names instance-attribute

entity_names = list(entity_names)

floating instance-attribute

floating = not root.has_immobile_base

device instance-attribute

device = root.device

link_names instance-attribute

link_names: list[str] = []

link_parent instance-attribute

link_parent = np.array([global_to_local.get(all_links[g].parent_idx, -1) for g in self._link_global], dtype=np.int64)

var_names instance-attribute

var_names: list[str] = ['base_x', 'base_y', 'base_z', 'base_rx', 'base_ry', 'base_rz'] if self.floating else []

n_v instance-attribute

n_v = len(self.var_names)

n_q instance-attribute

n_q = q_cursor

n_base_v instance-attribute

n_base_v = 6 if self.floating else 0

n_base_q instance-attribute

n_base_q = 7 if self.floating else 0

joint_moves_link instance-attribute

joint_moves_link = np.array([[self._joint_link[k] in chain[li] for k in range(len(self._joint_link))] for li in range(n_links)], dtype=bool).reshape(n_links, len(self._joint_link))

joint_lower instance-attribute

joint_lower = np.concatenate(lower)

joint_upper instance-attribute

joint_upper = np.concatenate(upper)

Methods:

entity_qpos

entity_qpos(q: ndarray) -> list[ndarray]

Split a rig configuration into each entity's qpos (dofs_name order).

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def entity_qpos(self, q: np.ndarray) -> list[np.ndarray]:
    """Split a rig configuration into each entity's ``qpos`` (dofs_name order)."""
    return [np.asarray(q)[..., s] for s in self._entity_q_slices]

set_configuration

set_configuration(q: ndarray) -> None

Write configuration q to the entities (runs Genesis forward kinematics).

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def set_configuration(self, q: np.ndarray) -> None:
    """Write configuration ``q`` to the entities (runs Genesis forward kinematics)."""
    # The floating joint rotates the rig about its qpos position, whatever point of the root link
    # Genesis anchors it at, so that point is the lever origin of the base rotation columns.
    self._base_pivot = np.array(q[0:3], dtype=np.float64) if self.floating else None
    for ent, qe in zip(self.entities, self.entity_qpos(q)):
        ent.set_qpos(torch.as_tensor(qe, dtype=gs.tc_float, device=self.device).unsqueeze(0))

link_poses

link_poses() -> tuple[ndarray, ndarray]

World (L, 3) positions and (L, 3, 3) orientations of every rig link at the set configuration.

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def link_poses(self) -> tuple[np.ndarray, np.ndarray]:
    """World ``(L, 3)`` positions and ``(L, 3, 3)`` orientations of every rig link at the set configuration."""
    pos, quat = [], []
    for ent in self.entities:
        pos.append(ent.get_links_pos(relative=False).reshape(-1, len(ent.links), 3)[0])
        quat.append(ent.get_links_quat(relative=False).reshape(-1, len(ent.links), 4)[0])
    pos = torch.cat(pos).cpu().numpy().astype(np.float64)
    quat = torch.cat(quat).cpu().numpy().astype(np.float64)
    return pos, quat_to_R(quat)

integrate

integrate(q: ndarray, v: ndarray) -> ndarray

Retract tangent step v onto configuration q (q (+) v).

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def integrate(self, q: np.ndarray, v: np.ndarray) -> np.ndarray:
    """Retract tangent step ``v`` onto configuration ``q`` (``q (+) v``)."""
    q = np.array(q, dtype=np.float64)
    v = np.asarray(v, dtype=np.float64)
    if self.floating:
        q[0:3] += v[0:3]
        quat = R_to_quat(quat_to_R(quat_from_axis_angle(v[3:6])) @ quat_to_R(q[3:7]))
        # The hemisphere of q, not whichever the matrix conversion lands in: the root quaternion of a retargeted
        # clip then never jumps q -> -q between frames (finite-difference angular velocity and slerp read that).
        q[3:7] = quat if quat @ q[3:7] >= 0.0 else -quat
    q[self.n_base_q :] += v[self.n_base_v :]
    return q

difference

difference(q1: ndarray, q0: ndarray) -> ndarray

Tangent v with q0 (+) v = q1 (q1 (-) q0).

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def difference(self, q1: np.ndarray, q0: np.ndarray) -> np.ndarray:
    """Tangent ``v`` with ``q0 (+) v = q1`` (``q1 (-) q0``)."""
    v = np.zeros(self.n_v)
    if self.floating:
        v[0:3] = q1[0:3] - q0[0:3]
        R1, R0 = quat_to_R(np.asarray(q1, dtype=np.float64)[3:7]), quat_to_R(np.asarray(q0, dtype=np.float64)[3:7])
        v[3:6] = axis_angle_from_quat(R_to_quat(R1 @ R0.T))
    v[self.n_base_v :] = np.asarray(q1)[self.n_base_q :] - np.asarray(q0)[self.n_base_q :]
    return v

link_index

link_index(name: str) -> int

Rig index of link name, raising with the available names if absent.

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def link_index(self, name: str) -> int:
    """Rig index of link ``name``, raising with the available names if absent."""
    if name not in self.link_names:
        raise KeyError(f"Link {name!r} is not in rig {self.entity_names}. Links: {self.link_names}.")
    return self.link_names.index(name)

link_jacobians

link_jacobians(link_pos: ndarray, link_rot: ndarray) -> ndarray

Spatial Jacobians of every link: (L, 6, n_v), rows [v of the link origin; omega] (world).

A point p on link l moves with J_v - [p - o_l]x J_omega, so a scalar residual with world gradient c at p has the row [c; (p - o_l) x c]^T J6_l: residuals on one link can be accumulated as a 6x6 block before touching the n_v columns.

Source code in eden/extensions/retargeting/otretarget/kinematics.py

def link_jacobians(self, link_pos: np.ndarray, link_rot: np.ndarray) -> np.ndarray:
    """Spatial Jacobians of every link: ``(L, 6, n_v)``, rows ``[v of the link origin; omega]`` (world).

    A point ``p`` on link ``l`` moves with ``J_v - [p - o_l]x J_omega``, so a scalar residual with
    world gradient ``c`` at ``p`` has the row ``[c; (p - o_l) x c]^T J6_l``: residuals on one link can
    be accumulated as a 6x6 block before touching the ``n_v`` columns.
    """
    n_links = len(self.link_names)
    jac = np.zeros((n_links, 6, self.n_v))
    if self.floating:
        jac[:, 0:3, 0:3] = np.eye(3)
        jac[:, 0:3, 3:6] = -skew(link_pos - self._base_pivot)
        jac[:, 3:6, 3:6] = np.eye(3)
    anchor, axis = self._joint_world(link_pos, link_rot)
    lever = link_pos[:, None, :] - anchor[None, :, :]  # (L, K, 3)
    lin = np.where(self._joint_revolute[None, :, None], np.cross(axis[None], lever), axis[None])
    ang = np.where(self._joint_revolute[None, :, None], axis[None], 0.0) * np.ones((n_links, 1, 1))
    moves = self.joint_moves_link[:, :, None]
    jac[:, 0:3, self._joint_var] = np.transpose(lin * moves, (0, 2, 1))
    jac[:, 3:6, self._joint_var] = np.transpose(ang * moves, (0, 2, 1))
    return jac

Functions:

Interaction channels: the surfaces probes read proximity triples (d, w, n) from (Sec. III-B).

A channel is a rigid surface — the ground or a manipulated object — with, in its own frame,

• a signed distance Phi(x) and a witness W(x) (the closest surface point), and
• a geodesic distance rho(w; w_hat) between two surface points (the placement residual r^x).

For a mesh, Phi and W are baked once on a voxel grid around the object (1 cm isotropic by default, reaching max_distance past the surface) and read back by trilinear interpolation, so a query costs no mesh search and the grid never needs recomputing as the object moves. For the ground, both are affine and exact. Geodesics are all-pairs shortest paths over the channel's surface point cloud, interpolated to arbitrary witness points through their nearest samples.

Classes

ChannelQuery dataclass

ChannelQuery(distance: ndarray, witness: ndarray, distance_grad: ndarray, witness_jac: ndarray, valid: ndarray)

Proximity of query points to a channel, all in the channel frame.

ATTRIBUTE	DESCRIPTION
distance	(N,) signed distance Phi(x) (negative inside). TYPE: ndarray
witness	(N, 3) witness W(x), the closest surface point. TYPE: ndarray
distance_grad	(N, 3) gradient of Phi. TYPE: ndarray
witness_jac	(N, 3, 3) Jacobian dW/dx. TYPE: ndarray
valid	(N,) bool, False where the point lies outside the baked region (then Phi >= max_distance is all that is known, and the other fields are meaningless). TYPE: ndarray

Attributes

distance instance-attribute

distance: ndarray

witness instance-attribute

witness: ndarray

distance_grad instance-attribute

distance_grad: ndarray

witness_jac instance-attribute

witness_jac: ndarray

valid instance-attribute

valid: ndarray

Methods:

Channel

Channel(name: str, max_distance: float)

Bases: ABC

A rigid surface probes measure proximity to.

PARAMETER	DESCRIPTION
name	The channel's name (the scene entity name for an object, "ground" for the floor). TYPE: str
max_distance	d_bar: the distance below which a probe-channel pair is active. TYPE: float

Source code in eden/extensions/retargeting/otretarget/channels.py

def __init__(self, name: str, max_distance: float) -> None:
    self.name = name
    self.max_distance = float(max_distance)

Attributes

name instance-attribute

name = name

max_distance instance-attribute

max_distance = float(max_distance)

Methods:

query abstractmethod

query(points: ndarray) -> ChannelQuery

Proximity of (N, 3) channel-frame points to the surface.

Source code in eden/extensions/retargeting/otretarget/channels.py

@abstractmethod
def query(self, points: np.ndarray) -> ChannelQuery:
    """Proximity of ``(N, 3)`` channel-frame points to the surface."""

geodesic abstractmethod

geodesic(w: ndarray, w_hat: ndarray, n_w: ndarray | None = None, n_hat: ndarray | None = None) -> tuple[ndarray, ndarray]

Geodesic distance rho(w; w_hat) between surface points, and its gradient in w.

PARAMETER	DESCRIPTION
w	(N, 3) channel-frame surface points. TYPE: ndarray
w_hat	(N, 3) channel-frame surface points. TYPE: ndarray
n_w	(N, 3) outward surface normals at w and w_hat: which face of a thin part each point is on. TYPE: ndarray DEFAULT: None
n_hat	(N, 3) outward surface normals at w and w_hat: which face of a thin part each point is on. TYPE: ndarray DEFAULT: None

RETURNS	DESCRIPTION
rho	(N,) distances. TYPE: ndarray
grad	(N, 3) gradient of rho with respect to w. TYPE: ndarray

Source code in eden/extensions/retargeting/otretarget/channels.py

@abstractmethod
def geodesic(
    self, w: np.ndarray, w_hat: np.ndarray, n_w: np.ndarray | None = None, n_hat: np.ndarray | None = None
) -> tuple[np.ndarray, np.ndarray]:
    """Geodesic distance ``rho(w; w_hat)`` between surface points, and its gradient in ``w``.

    Parameters
    ----------
    w, w_hat : np.ndarray
        ``(N, 3)`` channel-frame surface points.
    n_w, n_hat : np.ndarray, optional
        ``(N, 3)`` outward surface normals at ``w`` and ``w_hat``: which face of a thin part each point is on.

    Returns
    -------
    rho : np.ndarray
        ``(N,)`` distances.
    grad : np.ndarray
        ``(N, 3)`` gradient of ``rho`` with respect to ``w``.
    """

exact_distance abstractmethod

exact_distance(points: ndarray) -> tuple[ndarray, ndarray]

Exact (not grid-interpolated) signed distance and closest point, for evaluation.

Source code in eden/extensions/retargeting/otretarget/channels.py

@abstractmethod
def exact_distance(self, points: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    """Exact (not grid-interpolated) signed distance and closest point, for evaluation."""

exact_query abstractmethod

exact_query(points: ndarray) -> ChannelQuery

Like :meth:query, from the exact surface (for points beyond the baked region).

Source code in eden/extensions/retargeting/otretarget/channels.py

@abstractmethod
def exact_query(self, points: np.ndarray) -> ChannelQuery:
    """Like :meth:`query`, from the exact surface (for points beyond the baked region)."""

PlaneChannel

PlaneChannel(name: str, max_distance: float)

Bases: Channel

The ground: the plane z = 0 of its frame, with +z outside.

Phi(x) = x_z and W(x) = (x_x, x_y, 0) exactly, at any distance; the geodesic between two plane points is their Euclidean distance.

Source code in eden/extensions/retargeting/otretarget/channels.py

def __init__(self, name: str, max_distance: float) -> None:
    self.name = name
    self.max_distance = float(max_distance)

Methods:

query

query(points: ndarray) -> ChannelQuery

Source code in eden/extensions/retargeting/otretarget/channels.py

def query(self, points: np.ndarray) -> ChannelQuery:
    points = np.asarray(points, dtype=np.float64)
    n = len(points)
    witness = points.copy()
    witness[:, 2] = 0.0
    grad = np.zeros((n, 3))
    grad[:, 2] = 1.0
    jac = np.zeros((n, 3, 3))
    jac[:, 0, 0] = 1.0
    jac[:, 1, 1] = 1.0
    return ChannelQuery(points[:, 2].copy(), witness, grad, jac, np.ones(n, dtype=bool))

geodesic

geodesic(w: ndarray, w_hat: ndarray, n_w: ndarray | None = None, n_hat: ndarray | None = None) -> tuple[ndarray, ndarray]

Source code in eden/extensions/retargeting/otretarget/channels.py

def geodesic(
    self, w: np.ndarray, w_hat: np.ndarray, n_w: np.ndarray | None = None, n_hat: np.ndarray | None = None
) -> tuple[np.ndarray, np.ndarray]:
    delta = np.asarray(w, dtype=np.float64) - np.asarray(w_hat, dtype=np.float64)
    delta[:, 2] = 0.0
    rho = np.linalg.norm(delta, axis=1)
    grad = delta / np.maximum(rho, 1e-12)[:, None]
    return rho, grad

exact_distance

exact_distance(points: ndarray) -> tuple[ndarray, ndarray]

Source code in eden/extensions/retargeting/otretarget/channels.py

def exact_distance(self, points: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    points = np.asarray(points, dtype=np.float64)
    witness = points.copy()
    witness[:, 2] = 0.0
    return points[:, 2].copy(), witness

exact_query

exact_query(points: ndarray) -> ChannelQuery

Source code in eden/extensions/retargeting/otretarget/channels.py

def exact_query(self, points: np.ndarray) -> ChannelQuery:
    return self.query(points)

MeshChannel

MeshChannel(name: str, mesh: Trimesh, max_distance: float, resolution: float = 0.01, geodesic_density: float = 1000.0, solid_voxel: float = 0.0)

Bases: Channel

A rigid mesh, with Phi/W baked on a voxel grid in the mesh frame.

PARAMETER	DESCRIPTION
name	The channel's name. TYPE: str
mesh	The surface, in the channel (object) frame. The sign comes from the generalised winding number, which is only right on a closed, consistently oriented surface: on an open scan (a bin) it reads the empty cavity as solid and flips sign across the opening, so the distance jumps from +5 to -5 cm between two neighbouring points, and walls scanned as zero-thickness sheets have no inside at all. Pass solid_voxel for such a mesh (see :func:winding_sign_is_reliable and :func:watertight_solid). TYPE: Trimesh
max_distance	d_bar; the grid reaches this far past the mesh's bounding box. TYPE: float
resolution	Grid spacing (m). TYPE: float DEFAULT: 0.01
geodesic_density	Surface samples per m^2 for the geodesic table. TYPE: float DEFAULT: 1000.0
solid_voxel	If > 0 and the winding-number sign of mesh is not reliable (:func:winding_sign_is_reliable), the channel uses :func:watertight_solid of it at this voxel size (m) instead; source_mesh keeps the original. The solid lies one voxel outside the scan, so a scan that is already closed keeps its own surface. TYPE: float DEFAULT: 0.0

Source code in eden/extensions/retargeting/otretarget/channels.py

def __init__(
    self,
    name: str,
    mesh: trimesh.Trimesh,
    max_distance: float,
    resolution: float = 0.01,
    geodesic_density: float = 1000.0,
    solid_voxel: float = 0.0,
) -> None:
    super().__init__(name, max_distance)
    if not hasattr(igl, "FastWindingNumberBVH"):  # libigl < 2.6.3, as genesis-world installs it without the extra
        raise ImportError(
            f"MeshChannel '{name}' needs libigl >= 2.6.3 (igl.FastWindingNumberBVH, its winding-number tree); the "
            f"installed version is {installed_version('libigl') or 'not installed'}. Install the extra with "
            f"{install_hint('ot-retargeter')}; or upgrade alone with `uv pip install 'libigl>=2.6.3'`."
        )
    if resolution <= 0.0:
        raise ValueError(f"MeshChannel '{name}': resolution={resolution!r} must be > 0 (the paper uses 0.01 m).")
    if solid_voxel < 0.0:
        raise ValueError(
            f"MeshChannel '{name}': solid_voxel={solid_voxel!r} must be >= 0 (0 keeps the mesh as is)."
        )
    self.source_mesh = mesh
    if solid_voxel > 0.0 and not winding_sign_is_reliable(mesh):
        mesh = watertight_solid(mesh, solid_voxel)
    self.mesh = mesh
    self.resolution = float(resolution)
    self._vertices = np.ascontiguousarray(mesh.vertices, dtype=np.float64)
    self._faces = np.ascontiguousarray(mesh.faces, dtype=np.int64)
    # Search trees built once: igl.signed_distance rebuilds both on every call, 6x the cost of the query.
    self._aabb = igl.AABB()
    self._aabb.init(self._vertices, self._faces)
    self._winding = igl.FastWindingNumberBVH()
    self._winding.init(self._vertices, self._faces)

    margin = self.max_distance + 2.0 * self.resolution
    lo = self._vertices.min(axis=0) - margin
    hi = self._vertices.max(axis=0) + margin
    self.origin = lo
    self.shape = np.ceil((hi - lo) / self.resolution).astype(int) + 1
    axes = [self.origin[k] + self.resolution * np.arange(self.shape[k]) for k in range(3)]
    nodes = np.stack(np.meshgrid(*axes, indexing="ij"), axis=-1).reshape(-1, 3)
    distance, witness = self.exact_distance(nodes)
    self._phi = distance.reshape(*self.shape).astype(np.float32)
    self._witness = witness.reshape(*self.shape, 3).astype(np.float32)

    self.geodesics = SurfaceGeodesics.from_mesh(mesh, geodesic_density)

Attributes

source_mesh instance-attribute

source_mesh = mesh

mesh instance-attribute

mesh = mesh

resolution instance-attribute

resolution = float(resolution)

origin instance-attribute

origin = lo

shape instance-attribute

shape = np.ceil((hi - lo) / self.resolution).astype(int) + 1

geodesics instance-attribute

geodesics = SurfaceGeodesics.from_mesh(mesh, geodesic_density)

Methods:

exact_distance

exact_distance(points: ndarray) -> tuple[ndarray, ndarray]

Source code in eden/extensions/retargeting/otretarget/channels.py

def exact_distance(self, points: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    distance, _, witness = self._closest(points)
    return distance, witness

exact_query

exact_query(points: ndarray) -> ChannelQuery

Exact distance, witness and their derivatives.

The closest point of a triangle mesh slides with the query point only along the feature it lies on: dW/dx is the tangent projector I - n n^T inside a face, t t^T on an edge (t its direction) and zero at a vertex. Using the face projector everywhere overstates or misdirects the witness motion near edges, which a Gauss-Newton step then follows in the wrong direction.

Source code in eden/extensions/retargeting/otretarget/channels.py

def exact_query(self, points: np.ndarray) -> ChannelQuery:
    """Exact distance, witness and their derivatives.

    The closest point of a triangle mesh slides with the query point only along the feature it lies
    on: ``dW/dx`` is the tangent projector ``I - n n^T`` inside a face, ``t t^T`` on an edge (``t`` its
    direction) and zero at a vertex. Using the face projector everywhere overstates or misdirects the
    witness motion near edges, which a Gauss-Newton step then follows in the wrong direction.
    """
    points = np.ascontiguousarray(points, dtype=np.float64)
    distance, face, witness = self._closest(points)
    n = len(points)
    delta = points - witness
    length = np.linalg.norm(delta, axis=1)
    tri = self._vertices[self._faces[face]]  # (N, 3, 3)
    face_n = np.cross(tri[:, 1] - tri[:, 0], tri[:, 2] - tri[:, 0])
    face_n /= np.maximum(np.linalg.norm(face_n, axis=1, keepdims=True), 1e-12)
    sign = np.where(distance < 0, -1.0, 1.0)
    grad = np.where((length > 1e-9)[:, None], sign[:, None] * delta / np.maximum(length, 1e-12)[:, None], face_n)
    bary = trimesh.triangles.points_to_barycentric(tri, witness)
    on_zero = bary < 1e-6
    n_zero = on_zero.sum(axis=1)
    jac = np.eye(3)[None] - face_n[:, :, None] * face_n[:, None, :]
    edge = n_zero == 1
    if edge.any():
        # The edge joins the two vertices whose barycentric weight is non-zero.
        k = np.argmax(on_zero[edge], axis=1)
        a = tri[edge][np.arange(edge.sum()), (k + 1) % 3]
        b = tri[edge][np.arange(edge.sum()), (k + 2) % 3]
        t = (b - a) / np.maximum(np.linalg.norm(b - a, axis=1, keepdims=True), 1e-12)
        jac[edge] = t[:, :, None] * t[:, None, :]
    jac[n_zero >= 2] = 0.0
    return ChannelQuery(distance, witness, grad, jac, np.ones(n, dtype=bool))

query

query(points: ndarray) -> ChannelQuery

Source code in eden/extensions/retargeting/otretarget/channels.py

def query(self, points: np.ndarray) -> ChannelQuery:
    points = np.asarray(points, dtype=np.float64)
    u = (points - self.origin) / self.resolution
    upper = self.shape - 1
    valid = np.all((u >= 0.0) & (u <= upper), axis=1)
    i0 = np.clip(np.floor(u).astype(int), 0, upper - 1)
    f = np.clip(u - i0, 0.0, 1.0)

    n = len(points)
    distance = np.zeros(n)
    grad = np.zeros((n, 3))
    witness = np.zeros((n, 3))
    jac = np.zeros((n, 3, 3))
    for corner in range(8):
        bits = np.array([(corner >> 2) & 1, (corner >> 1) & 1, corner & 1])
        idx = i0 + bits
        # Trilinear weight and its derivative along each axis.
        factors = np.where(bits[None, :] == 1, f, 1.0 - f)
        dfactors = np.where(bits == 1, 1.0, -1.0)
        weight = factors.prod(axis=1)
        dweight = (
            np.stack(
                [
                    dfactors[0] * factors[:, 1] * factors[:, 2],
                    dfactors[1] * factors[:, 0] * factors[:, 2],
                    dfactors[2] * factors[:, 0] * factors[:, 1],
                ],
                axis=1,
            )
            / self.resolution
        )
        phi_c = self._phi[idx[:, 0], idx[:, 1], idx[:, 2]].astype(np.float64)
        w_c = self._witness[idx[:, 0], idx[:, 1], idx[:, 2]].astype(np.float64)
        distance += weight * phi_c
        grad += dweight * phi_c[:, None]
        witness += weight[:, None] * w_c
        jac += w_c[:, :, None] * dweight[:, None, :]
    return ChannelQuery(distance, witness, grad, jac, valid)

geodesic

geodesic(w: ndarray, w_hat: ndarray, n_w: ndarray | None = None, n_hat: ndarray | None = None, hat: tuple[ndarray, ndarray] | None = None) -> tuple[ndarray, ndarray]

See :meth:Channel.geodesic; hat is :meth:SurfaceGeodesics.endpoint_weights at w_hat, if known.

Source code in eden/extensions/retargeting/otretarget/channels.py

def geodesic(
    self,
    w: np.ndarray,
    w_hat: np.ndarray,
    n_w: np.ndarray | None = None,
    n_hat: np.ndarray | None = None,
    hat: tuple[np.ndarray, np.ndarray] | None = None,
) -> tuple[np.ndarray, np.ndarray]:
    """See :meth:`Channel.geodesic`; ``hat`` is :meth:`SurfaceGeodesics.endpoint_weights` at ``w_hat``, if known."""
    return self.geodesics.distance(w, w_hat, n_w, n_hat, hat=hat)

SurfaceGeodesics

SurfaceGeodesics(points: ndarray, normals: ndarray, spacing: float, k: int = 10)

All-pairs geodesic distances over a surface point cloud, interpolated to arbitrary surface points.

The cloud is connected into a graph by k-nearest-neighbour edges between samples whose normals agree (so a thin plate's two faces are not short-circuited through its thickness); components left disconnected — separate shells of one object — are bridged at their closest pair. Distances between samples are Dijkstra shortest paths.

For two arbitrary surface points the distance is interpolated continuously — the solver's Gauss-Newton model must describe the cost it is accepted against, and a nearest-sample minimum jumps whenever the nearest set changes. Each endpoint spreads over the samples within support * spacing with compactly supported weights (1 - r / R)^2; the distance is the sample table interpolated at both ends, sum_ab alpha_a(w) alpha_b(w_hat) G[a, b]; and, for endpoints a few spacings apart, it blends by a smoothstep into the straight segment, which is what a geodesic reduces to locally.

PARAMETER	DESCRIPTION
points	(N, 3) surface samples. TYPE: ndarray
normals	(N, 3) unit outward normals at the samples. TYPE: ndarray
spacing	The mean sample spacing (m). TYPE: float
k	Graph neighbours per sample. TYPE: int DEFAULT: 10

Source code in eden/extensions/retargeting/otretarget/channels.py

def __init__(self, points: np.ndarray, normals: np.ndarray, spacing: float, k: int = 10) -> None:
    self.points = np.asarray(points, dtype=np.float64)
    self.normals = np.asarray(normals, dtype=np.float64)
    self.spacing = float(spacing)
    n = len(self.points)
    self._tree = cKDTree(self.points)
    # The k nearest samples *of the same facing*: on a part thinner than the spacing about half the plain
    # nearest neighbours lie on the opposite face, and dropping them left the face so sparsely connected that
    # paths across it ran up to 10x the straight line.
    k_eff = min(4 * k + 1, n)
    dist, nbr = self._tree.query(self.points, k=k_eff)
    dist, nbr = dist[:, 1:], nbr[:, 1:]
    ok = (np.einsum("na,nka->nk", self.normals, self.normals[nbr]) > -0.5) & (dist <= 4.0 * self.spacing)
    ok &= np.cumsum(ok, axis=1) <= k
    rows = np.repeat(np.arange(n), nbr.shape[1])[ok.reshape(-1)]
    cols, lengths = nbr[ok], dist[ok]
    rows, cols, lengths = list(rows), list(cols), list(lengths)

    # Bridge disconnected shells at their closest pair until the graph is connected.
    while True:
        graph = coo_matrix((lengths, (rows, cols)), shape=(n, n)).tocsr()
        n_comp, labels = connected_components(graph, directed=False)
        if n_comp == 1:
            break
        sizes = np.bincount(labels)
        comp = int(np.argmin(sizes))
        inside = np.flatnonzero(labels == comp)
        outside = np.flatnonzero(labels != comp)
        d_out, j_out = cKDTree(self.points[outside]).query(self.points[inside])
        best = int(np.argmin(d_out))
        rows.append(int(inside[best]))
        cols.append(int(outside[j_out[best]]))
        lengths.append(float(max(d_out[best], 1e-9)))
    self.table = shortest_path(graph, method="D", directed=False).astype(np.float32)

Attributes

points instance-attribute

points = np.asarray(points, dtype=np.float64)

normals instance-attribute

normals = np.asarray(normals, dtype=np.float64)

spacing instance-attribute

spacing = float(spacing)

table instance-attribute

table = shortest_path(graph, method='D', directed=False).astype(np.float32)

tree property

tree: cKDTree

k-d tree of the samples.

Methods:

from_mesh classmethod

from_mesh(mesh: Trimesh, density: float, seed: int = 0) -> SurfaceGeodesics

Sample mesh at density points per m^2 and build the geodesic table on the samples.

Source code in eden/extensions/retargeting/otretarget/channels.py

@classmethod
def from_mesh(cls, mesh: trimesh.Trimesh, density: float, seed: int = 0) -> SurfaceGeodesics:
    """Sample ``mesh`` at ``density`` points per m^2 and build the geodesic table on the samples."""
    n = max(int(np.ceil(mesh.area * density)), 16)
    points, faces = trimesh.sample.sample_surface_even(mesh, n, seed=seed)
    if len(points) < 16:
        points, faces = trimesh.sample.sample_surface(mesh, n, seed=seed)
    return cls(points, mesh.face_normals[faces], spacing=1.0 / np.sqrt(density))

endpoint_weights

endpoint_weights(x: ndarray, normal: ndarray | None = None, k: int = 16, support: float = 1.5) -> tuple[ndarray, ndarray]

Neighbour indices (M, k) and normalised weights (M, k) of the table at the surface points x.

This is the w_hat end of :meth:distance. Each row depends on its own point only, so a caller whose targets stay fixed over several :meth:distance calls computes it once and passes it as hat.

Source code in eden/extensions/retargeting/otretarget/channels.py

def endpoint_weights(
    self, x: np.ndarray, normal: np.ndarray | None = None, k: int = 16, support: float = 1.5
) -> tuple[np.ndarray, np.ndarray]:
    """Neighbour indices ``(M, k)`` and normalised weights ``(M, k)`` of the table at the surface points ``x``.

    This is the ``w_hat`` end of :meth:`distance`. Each row depends on its own point only, so a caller whose
    targets stay fixed over several :meth:`distance` calls computes it once and passes it as ``hat``.
    """
    idx, _, alpha, _, _ = self._weights(np.asarray(x, dtype=np.float64), k, support, normal)
    return idx, alpha

distance

distance(w: ndarray, w_hat: ndarray, n_w: ndarray | None = None, n_hat: ndarray | None = None, k: int = 16, support: float = 1.5, blend: tuple[float, float] = (2.0, 4.0), soften: float = 0.05, hat: tuple[ndarray, ndarray] | None = None) -> tuple[ndarray, ndarray]

Interpolated geodesic distance and its gradient in w (see the class docstring).

PARAMETER	DESCRIPTION
w	(M, 3) surface points. TYPE: ndarray
w_hat	(M, 3) surface points. TYPE: ndarray
n_w	(M, 3) outward normals at w and w_hat; each endpoint then interpolates over the samples of its own face only (see :meth:_weights). Without them a point near a thin part's rim blends both faces. TYPE: ndarray DEFAULT: None
n_hat	(M, 3) outward normals at w and w_hat; each endpoint then interpolates over the samples of its own face only (see :meth:_weights). Without them a point near a thin part's rim blends both faces. TYPE: ndarray DEFAULT: None
k	Neighbours examined per endpoint (must cover the support). TYPE: int DEFAULT: 16
support	Weight support radius, in sample spacings. TYPE: float DEFAULT: 1.5
soften	eps of the softened end normals (see the comment in the body): bounds the agreement's gradient by 1 / eps where the interpolated normal cancels, and shifts it little elsewhere. TYPE: float DEFAULT: 0.05
blend	Chord lengths (in spacings) over which the distance blends from the straight segment (below) to the graph distance (above). TYPE: tuple[float, float] DEFAULT: (2.0, 4.0)
hat	:meth:endpoint_weights at w_hat (with n_hat, k and support), computed here if None. TYPE: tuple[ndarray, ndarray] DEFAULT: None

Source code in eden/extensions/retargeting/otretarget/channels.py

def distance(
    self,
    w: np.ndarray,
    w_hat: np.ndarray,
    n_w: np.ndarray | None = None,
    n_hat: np.ndarray | None = None,
    k: int = 16,
    support: float = 1.5,
    blend: tuple[float, float] = (2.0, 4.0),
    soften: float = 0.05,
    hat: tuple[np.ndarray, np.ndarray] | None = None,
) -> tuple[np.ndarray, np.ndarray]:
    """Interpolated geodesic distance and its gradient in ``w`` (see the class docstring).

    Parameters
    ----------
    w, w_hat : np.ndarray
        ``(M, 3)`` surface points.
    n_w, n_hat : np.ndarray, optional
        ``(M, 3)`` outward normals at ``w`` and ``w_hat``; each endpoint then interpolates over the samples of
        its own face only (see :meth:`_weights`). Without them a point near a thin part's rim blends both faces.
    k : int
        Neighbours examined per endpoint (must cover the support).
    support : float
        Weight support radius, in sample spacings.
    soften : float
        ``eps`` of the softened end normals (see the comment in the body): bounds the agreement's gradient by
        ``1 / eps`` where the interpolated normal cancels, and shifts it little elsewhere.
    blend : tuple[float, float]
        Chord lengths (in spacings) over which the distance blends from the straight segment (below)
        to the graph distance (above).
    hat : tuple[np.ndarray, np.ndarray], optional
        :meth:`endpoint_weights` at ``w_hat`` (with ``n_hat``, ``k`` and ``support``), computed here if None.
    """
    w = np.asarray(w, dtype=np.float64)
    w_hat = np.asarray(w_hat, dtype=np.float64)
    ia, _, aa, daa, _ = self._weights(w, k, support, n_w)
    ib, ab = self.endpoint_weights(w_hat, n_hat, k, support) if hat is None else hat
    # Interpolated outward normals at both ends; the straight segment is a valid local geodesic only on a
    # consistently oriented patch (not across a thin plate, whose faces are close but opposite). Each is divided by
    # sqrt(|sum alpha n|^2 + eps^2), not its length: at the narrow end of a thin slat the interpolation averages the
    # slat's two opposite sides to nearly nothing (|sum alpha n| = 4e-6), and divided by that the agreement
    # gradient reached 8e5. The softened normal is short there, so the agreement is small and the graph distance
    # is used. (Holding the caller's normals fixed instead left the gradient inexact wherever the agreement blends,
    # and a cold first frame wandered off.)
    eps = soften
    raw_w = (aa[:, :, None] * self.normals[ia]).sum(axis=1)
    raw_h = (ab[:, :, None] * self.normals[ib]).sum(axis=1)
    len_w = np.sqrt((raw_w * raw_w).sum(axis=1) + eps * eps)
    unit_w = raw_w / len_w[:, None]
    unit_h = raw_h / np.sqrt((raw_h * raw_h).sum(axis=1) + eps * eps)[:, None]
    agree = (unit_w * unit_h).sum(axis=1)
    # d(agree)/dw = ((I - u u^T) / L) (sum_a n_a dalpha_a^T) applied to n_h,
    # with u = raw / L, L = sqrt(|raw|^2 + eps^2)
    draw = np.einsum("mka,mkb->mab", self.normals[ia], daa)  # d raw_w / dw
    proj_h = unit_h - agree[:, None] * unit_w
    d_agree = np.einsum("ma,mab->mb", proj_h, draw) / len_w[:, None]
    ta = np.clip((agree - 0.2) / 0.4, 0.0, 1.0)
    h = ta * ta * (3.0 - 2.0 * ta)
    dh = np.where((agree > 0.2) & (agree < 0.6), 6.0 * ta * (1.0 - ta) / 0.4, 0.0)[:, None] * d_agree
    # The sample table interpolated at both endpoints: exact at the samples, smooth in between.
    to_b = (self.table[ia[:, :, None], ib[:, None, :]] * ab[:, None, :]).sum(axis=2)  # (M, k)
    graph = (aa * to_b).sum(axis=1)
    g_graph = (daa * to_b[:, :, None]).sum(axis=1)

    chord_vec = w - w_hat
    chord = np.linalg.norm(chord_vec, axis=1)
    g_chord = chord_vec / np.maximum(chord, 1e-12)[:, None]
    lo, hi = blend[0] * self.spacing, blend[1] * self.spacing
    tt = np.clip((chord - lo) / (hi - lo), 0.0, 1.0)
    near = 1.0 - tt * tt * (3.0 - 2.0 * tt)  # 1 for a short chord, 0 beyond ``blend[1]``
    dnear = -np.where((chord > lo) & (chord < hi), 6.0 * tt * (1.0 - tt) / (hi - lo), 0.0)[:, None] * g_chord
    # Weight of the straight segment: a short chord between agreeing normals.
    c = near * h
    dc = dnear * h[:, None] + near[:, None] * dh
    rho = c * chord + (1.0 - c) * graph
    grad = c[:, None] * g_chord + (1.0 - c)[:, None] * g_graph + (chord - graph)[:, None] * dc
    return rho, grad

Functions:

winding_sign_is_reliable

winding_sign_is_reliable(mesh: Trimesh, offset: float = 0.0005, samples: int = 4000) -> bool

Whether the generalised winding number of mesh gives the right inside/outside sign at its surface.

True when, with duplicate vertices merged, the surface has no boundary edge and a consistent orientation, and at least 98% of surface samples read inside offset behind their face and outside offset in front of it. A scan stored as loose triangles (every triangle with its own vertices, as the OMOMO objects are) is closed once merged and passes; Trimesh.is_watertight on the unmerged mesh reads it as open. An open bin, or walls scanned as zero-thickness sheets (no inside behind the face), fail.

Source code in eden/extensions/retargeting/otretarget/channels.py

def winding_sign_is_reliable(mesh: trimesh.Trimesh, offset: float = 5e-4, samples: int = 4000) -> bool:
    """Whether the generalised winding number of ``mesh`` gives the right inside/outside sign at its surface.

    True when, with duplicate vertices merged, the surface has no boundary edge and a consistent orientation, and at
    least 98% of surface samples read inside ``offset`` behind their face and outside ``offset`` in front of it. A
    scan stored as loose triangles (every triangle with its own vertices, as the OMOMO objects are) is closed once
    merged and passes; ``Trimesh.is_watertight`` on the unmerged mesh reads it as open. An open bin, or walls scanned
    as zero-thickness sheets (no inside behind the face), fail.
    """
    merged = trimesh.Trimesh(np.asarray(mesh.vertices), np.asarray(mesh.faces), process=True)
    _, uses = np.unique(np.sort(merged.edges, axis=1), axis=0, return_counts=True)
    if (uses == 1).any() or not merged.is_winding_consistent:
        return False
    points, face = trimesh.sample.sample_surface(mesh, samples, seed=0)
    normal = mesh.face_normals[face]
    vertices = np.ascontiguousarray(mesh.vertices, dtype=np.float64)
    faces = np.ascontiguousarray(mesh.faces, dtype=np.int64)
    inside = np.asarray(igl.fast_winding_number(vertices, faces, points - offset * normal)) > 0.5
    outside = np.asarray(igl.fast_winding_number(vertices, faces, points + offset * normal)) < 0.5
    return bool(inside.mean() >= 0.98 and outside.mean() >= 0.98)

watertight_solid

watertight_solid(mesh: Trimesh, voxel: float, cache: bool = True) -> Trimesh

A closed surface around the solid an open or fragmented scan stands for.

Space within voxel of the scan is a shell (a single-sheet wall gets that thickness); free space reachable from outside the bounding box without crossing the shell is outside, so the cavity of an open bin stays free space, while space the shell encloses (the inside of a closed box) is solid. The surface is the zero level of distance - voxel over outside and shell space and -distance inside, extracted by marching cubes on a grid of spacing voxel: closed, and voxel outside the scanned surface. The result is cached by mesh content.

Source code in eden/extensions/retargeting/otretarget/channels.py

def watertight_solid(mesh: trimesh.Trimesh, voxel: float, cache: bool = True) -> trimesh.Trimesh:
    """A closed surface around the solid an open or fragmented scan stands for.

    Space within ``voxel`` of the scan is a shell (a single-sheet wall gets that thickness); free space reachable
    from outside the bounding box without crossing the shell is outside, so the cavity of an open bin stays free
    space, while space the shell encloses (the inside of a closed box) is solid. The surface is the zero level of
    ``distance - voxel`` over outside and shell space and ``-distance`` inside, extracted by marching cubes on a grid
    of spacing ``voxel``: closed, and ``voxel`` outside the scanned surface. The result is cached by mesh content.
    """
    import hashlib

    from scipy import ndimage

    vertices = np.ascontiguousarray(mesh.vertices, dtype=np.float64)
    faces = np.ascontiguousarray(mesh.faces, dtype=np.int64)
    key = hashlib.sha1(vertices.tobytes() + faces.tobytes() + np.float64(voxel).tobytes()).hexdigest()[:20]
    path = None
    if cache:
        from eden.utils.misc import get_cache_dir

        folder = get_cache_dir("otretarget", "watertight")
        os.makedirs(folder, exist_ok=True)
        path = os.path.join(folder, f"{key}.npz")
        if os.path.exists(path):
            data = np.load(path)
            return trimesh.Trimesh(data["vertices"], data["faces"], process=False)
    try:
        from skimage.measure import marching_cubes
    except ImportError as exc:  # pragma: no cover - exercised only without the optional dependency
        raise ImportError(
            "watertight_solid extracts the closed surface with scikit-image: install it with `pip install "
            "scikit-image` (the `ot-retargeter` extra of eden)."
        ) from exc
    pad = 4.0 * voxel
    lo = vertices.min(axis=0) - pad
    shape = np.ceil((vertices.max(axis=0) + pad - lo) / voxel).astype(int) + 1
    axes = [lo[k] + voxel * np.arange(shape[k]) for k in range(3)]
    tree = igl.AABB()
    tree.init(vertices, faces)
    # Slab by slab along x: the full node array of a 1.7 m lamp at 4 mm is 12 M points (300 MB in float64).
    yz = np.stack(np.meshgrid(axes[1], axes[2], indexing="ij"), axis=-1).reshape(-1, 2)
    distance = np.empty(shape, dtype=np.float32)
    for i, x in enumerate(axes[0]):
        slab = np.concatenate([np.full((len(yz), 1), x), yz], axis=1)
        distance[i] = np.sqrt(tree.squared_distance(vertices, faces, slab)[0]).reshape(shape[1], shape[2])
    free = distance >= voxel
    labels, _ = ndimage.label(free)
    rim = np.concatenate([labels[[0, -1]].ravel(), labels[:, [0, -1]].ravel(), labels[:, :, [0, -1]].ravel()])
    outside = np.isin(labels, np.unique(rim[rim > 0]))
    field = np.where(outside | ~free, distance - voxel, -distance)
    verts, tris, _, _ = marching_cubes(field, level=0.0, spacing=(voxel, voxel, voxel))
    solid = trimesh.Trimesh(verts + lo, tris, process=True)
    solid.fix_normals()
    # Closed means no boundary edge: every edge on an even number of faces. Marching cubes puts an edge on four
    # faces where two cells touch only along it, which ``is_watertight`` (exactly two) refuses but which encloses.
    _, uses = np.unique(np.sort(solid.edges, axis=1), axis=0, return_counts=True)
    if (uses % 2).any():
        raise RuntimeError(
            f"watertight_solid: the extracted surface of a {len(faces)}-face mesh has {int((uses % 2).sum())} boundary "
            f"edges at voxel={voxel}; a surface strictly inside the grid has none -- report the mesh."
        )
    if path is not None:
        tmp = f"{path}.{os.getpid()}.tmp.npz"
        np.savez_compressed(tmp, vertices=solid.vertices, faces=solid.faces)
        os.replace(tmp, path)
    return solid

Probe clouds: dense surface samples carried by the human, the robot links and the objects (Sec. III-B).

Every entity carries a probe cloud, sampled uniformly by area. Robot and object probes are rigid in their carrier's frame (a link, an object). Human probes ride a deforming mesh, so they are stored as barycentric coordinates on its triangles and re-posed every frame, each labelled with the body part (the heaviest skinning joint) of the triangle it lies on.

Classes

RigidProbes dataclass

RigidProbes(local: ndarray, normals: ndarray, carrier: ndarray, area: ndarray)

Probes rigidly attached to carriers (robot links or objects).

ATTRIBUTE	DESCRIPTION
local	(N, 3) positions in the carrier frame. TYPE: ndarray
normals	(N, 3) outward surface normals in the carrier frame. TYPE: ndarray
carrier	(N,) carrier index (rig link index, or object index). TYPE: ndarray
area	(N,) surface area each probe stands for (its carrier's area over its probe count), m^2. TYPE: ndarray

Attributes

local instance-attribute

local: ndarray

normals instance-attribute

normals: ndarray

carrier instance-attribute

carrier: ndarray

area instance-attribute

area: ndarray

Methods:

world

world(carrier_pos: ndarray, carrier_rot: ndarray) -> ndarray

(N, 3) world positions given carrier poses (C, 3), (C, 3, 3).

Source code in eden/extensions/retargeting/otretarget/probes.py

def world(self, carrier_pos: np.ndarray, carrier_rot: np.ndarray) -> np.ndarray:
    """``(N, 3)`` world positions given carrier poses ``(C, 3)``, ``(C, 3, 3)``."""
    return carrier_pos[self.carrier] + np.einsum("nab,nb->na", carrier_rot[self.carrier], self.local)

HumanProbes dataclass

HumanProbes(corners: ndarray, bary: ndarray, part: ndarray, area: ndarray)

Probes on the human mesh, as barycentric coordinates on its triangles.

ATTRIBUTE	DESCRIPTION
corners	(N, 3) vertex indices of each probe's triangle. TYPE: ndarray
bary	(N, 3) barycentric weights. TYPE: ndarray
part	(N,) index of the skeleton joint whose part the probe lies on. TYPE: ndarray
area	(N,) rest-surface area each probe stands for (its part's area over its probe count), m^2. TYPE: ndarray

Attributes

corners instance-attribute

corners: ndarray

bary instance-attribute

bary: ndarray

part instance-attribute

part: ndarray

area instance-attribute

area: ndarray

Methods:

positions

positions(vertices: ndarray) -> ndarray

Probe positions on a posed mesh: (V, 3) -> (N, 3) or (T, V, 3) -> (T, N, 3).

Source code in eden/extensions/retargeting/otretarget/probes.py

def positions(self, vertices: np.ndarray) -> np.ndarray:
    """Probe positions on a posed mesh: ``(V, 3) -> (N, 3)`` or ``(T, V, 3) -> (T, N, 3)``."""
    return np.einsum("...nkc,nk->...nc", vertices[..., self.corners, :], self.bary)

normals

normals(vertices: ndarray) -> ndarray

Unit outward normals of the probes' triangles on a posed (V, 3) mesh: (N, 3).

Source code in eden/extensions/retargeting/otretarget/probes.py

def normals(self, vertices: np.ndarray) -> np.ndarray:
    """Unit outward normals of the probes' triangles on a posed ``(V, 3)`` mesh: ``(N, 3)``."""
    tri = np.asarray(vertices, dtype=np.float64)[self.corners]
    n = np.cross(tri[:, 1] - tri[:, 0], tri[:, 2] - tri[:, 0])
    return n / np.maximum(np.linalg.norm(n, axis=1, keepdims=True), 1e-12)

Functions:

resolve_density

resolve_density(name: str, densities: Mapping[str, float]) -> float

The density of the first pattern in densities that fully matches name.

PARAMETER	DESCRIPTION
name	A link or body-part name. TYPE: str
densities	Ordered {regex: points per m^2}; put specific patterns before a catch-all ".*". TYPE: Mapping[str, float]

Source code in eden/extensions/retargeting/otretarget/probes.py

def resolve_density(name: str, densities: Mapping[str, float]) -> float:
    """The density of the first pattern in ``densities`` that fully matches ``name``.

    Parameters
    ----------
    name : str
        A link or body-part name.
    densities : Mapping[str, float]
        Ordered ``{regex: points per m^2}``; put specific patterns before a catch-all ``".*"``.
    """
    for pattern, density in densities.items():
        if re.fullmatch(pattern, name):
            return float(density)
    raise ValueError(
        f"No probe density pattern matches {name!r}. Patterns: {list(densities)}. Add a catch-all '.*' entry."
    )

sample_surface

sample_surface(mesh: Trimesh, density: float, min_points: int, seed: int = 0) -> tuple[ndarray, ndarray, ndarray, ndarray]

Area-uniform samples of a mesh surface at density points per m^2 (at least min_points).

RETURNS	DESCRIPTION
points	(N, 3). TYPE: ndarray
normals	(N, 3) face normals at the samples. TYPE: ndarray
faces	(N,) face index of each sample. TYPE: ndarray
bary	(N, 3) barycentric coordinates in that face. TYPE: ndarray

Source code in eden/extensions/retargeting/otretarget/probes.py

def sample_surface(
    mesh: trimesh.Trimesh, density: float, min_points: int, seed: int = 0
) -> tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray]:
    """Area-uniform samples of a mesh surface at ``density`` points per m^2 (at least ``min_points``).

    Returns
    -------
    points : np.ndarray
        ``(N, 3)``.
    normals : np.ndarray
        ``(N, 3)`` face normals at the samples.
    faces : np.ndarray
        ``(N,)`` face index of each sample.
    bary : np.ndarray
        ``(N, 3)`` barycentric coordinates in that face.
    """
    n = max(int(np.ceil(mesh.area * density)), int(min_points))
    points, faces = trimesh.sample.sample_surface_even(mesh, n, seed=seed)
    if len(points) < max(min_points, n // 2):
        # Even sampling rejects aggressively on tiny or sliver-heavy meshes; fall back to plain area sampling.
        points, faces = trimesh.sample.sample_surface(mesh, n, seed=seed)
    bary = trimesh.triangles.points_to_barycentric(mesh.triangles[faces], points)
    return np.asarray(points), mesh.face_normals[faces], np.asarray(faces), bary

sample_rig_probes

sample_rig_probes(kin: RigKinematics, densities: Mapping[str, float], min_points: int, exclude: list[str] | None = None, use_visual_mesh: bool = True, seed: int = 0) -> RigidProbes

Sample every rig link that has geometry, in its link frame.

PARAMETER	DESCRIPTION
kin	The rig. TYPE: RigKinematics
densities	{link-name regex: points per m^2}. TYPE: Mapping[str, float]
min_points	Floor on samples per link, so small links (finger phalanges) still carry probes. TYPE: int
exclude	Link-name regexes that carry no probes. TYPE: list[str] DEFAULT: None
use_visual_mesh	Sample the visual meshes (the paper's choice) rather than the collision geometry. TYPE: bool DEFAULT: True
seed	Sampling seed. TYPE: int DEFAULT: 0

Source code in eden/extensions/retargeting/otretarget/probes.py

def sample_rig_probes(
    kin: RigKinematics,
    densities: Mapping[str, float],
    min_points: int,
    exclude: list[str] | None = None,
    use_visual_mesh: bool = True,
    seed: int = 0,
) -> RigidProbes:
    """Sample every rig link that has geometry, in its link frame.

    Parameters
    ----------
    kin : RigKinematics
        The rig.
    densities : Mapping[str, float]
        ``{link-name regex: points per m^2}``.
    min_points : int
        Floor on samples per link, so small links (finger phalanges) still carry probes.
    exclude : list[str], optional
        Link-name regexes that carry no probes.
    use_visual_mesh : bool
        Sample the visual meshes (the paper's choice) rather than the collision geometry.
    seed : int
        Sampling seed.
    """
    exclude = exclude or []
    links = {link.name: link for ent in kin.entities for link in ent.links}
    local, normals, carrier, area = [], [], [], []
    for li, name in enumerate(kin.link_names):
        if any(re.fullmatch(p, name) for p in exclude):
            continue
        geoms = links[name].vgeoms if use_visual_mesh else links[name].geoms
        if len(geoms) == 0:
            continue
        mesh = get_link_mesh(links[name], use_visual_mesh=use_visual_mesh)
        if mesh.area <= 1e-8:
            continue
        pts, nrm, _, _ = sample_surface(mesh, resolve_density(name, densities), min_points, seed=seed + li)
        local.append(pts)
        normals.append(nrm)
        carrier.append(np.full(len(pts), li, dtype=np.int64))
        area.append(np.full(len(pts), mesh.area / len(pts)))
    if not local:
        raise ValueError(f"sample_rig_probes: no link of rig {kin.entity_names} has geometry to sample.")
    return RigidProbes(np.concatenate(local), np.concatenate(normals), np.concatenate(carrier), np.concatenate(area))

sample_object_probes

sample_object_probes(meshes: list[Trimesh], density: float, min_points: int, seed: int = 0) -> RigidProbes

Sample each object mesh in its own frame; the carrier index is the object's position in meshes.

Source code in eden/extensions/retargeting/otretarget/probes.py

def sample_object_probes(meshes: list[trimesh.Trimesh], density: float, min_points: int, seed: int = 0) -> RigidProbes:
    """Sample each object mesh in its own frame; the carrier index is the object's position in ``meshes``."""
    local, normals, carrier, area = [], [], [], []
    for oi, mesh in enumerate(meshes):
        pts, nrm, _, _ = sample_surface(mesh, density, min_points, seed=seed + oi)
        local.append(pts)
        normals.append(nrm)
        carrier.append(np.full(len(pts), oi, dtype=np.int64))
        area.append(np.full(len(pts), mesh.area / len(pts)))
    if not local:
        return RigidProbes(np.zeros((0, 3)), np.zeros((0, 3)), np.zeros(0, dtype=np.int64), np.zeros(0))
    return RigidProbes(np.concatenate(local), np.concatenate(normals), np.concatenate(carrier), np.concatenate(area))

sample_human_probes

sample_human_probes(human: HumanSurfaceMotion, densities: Mapping[str, float], min_points: int, seed: int = 0) -> HumanProbes

Sample the human rest mesh part by part, each part at the density its joint name resolves to.

A triangle belongs to the part of the joint with the largest summed skinning weight over its three vertices.

PARAMETER	DESCRIPTION
human	The motion (its rest mesh is sampled). TYPE: HumanSurfaceMotion
densities	{joint-name regex: points per m^2}. TYPE: Mapping[str, float]
min_points	Floor on samples per part. TYPE: int
seed	Sampling seed. TYPE: int DEFAULT: 0

Source code in eden/extensions/retargeting/otretarget/probes.py

def sample_human_probes(
    human: HumanSurfaceMotion, densities: Mapping[str, float], min_points: int, seed: int = 0
) -> HumanProbes:
    """Sample the human rest mesh part by part, each part at the density its joint name resolves to.

    A triangle belongs to the part of the joint with the largest summed skinning weight over its
    three vertices.

    Parameters
    ----------
    human : HumanSurfaceMotion
        The motion (its rest mesh is sampled).
    densities : Mapping[str, float]
        ``{joint-name regex: points per m^2}``.
    min_points : int
        Floor on samples per part.
    seed : int
        Sampling seed.
    """
    faces = np.asarray(human.faces)
    face_part = human.skin_weights[faces].sum(axis=1).argmax(axis=1)
    corners, bary, part, area = [], [], [], []
    for j, name in enumerate(human.joint_names):
        mask = face_part == j
        if not mask.any():
            continue
        sub_faces = faces[mask]
        mesh = trimesh.Trimesh(human.rest_vertices, sub_faces, process=False)
        _, _, fidx, b = sample_surface(mesh, resolve_density(name, densities), min_points, seed=seed + j)
        corners.append(sub_faces[fidx])
        bary.append(b)
        part.append(np.full(len(fidx), j, dtype=np.int64))
        area.append(np.full(len(fidx), mesh.area / len(fidx)))
    return HumanProbes(np.concatenate(corners), np.concatenate(bary), np.concatenate(part), np.concatenate(area))

Entropic optimal transport between surface point clouds (OTRetarget Sec. III-B, Eq. 3-4).

A demonstration's interaction targets live on the human (or demonstrated-object) surface, the residuals on the robot (or substitute-object) surface. The correspondence between the two is the argmax of an entropically regularised transport plan between the two clouds, each centred and scaled to unit root-mean-square radius so a size mismatch does not decide the matching. Unlike a nearest-neighbour assignment, the marginal constraints force every source sample to receive mass, so limb extremities (fingertips, heels) are not left unmatched.

The paper's cost is the squared distance between the normalised clouds. With outward normals given, the cost also carries normal_weight * |n_i - n_p|^2: a robot phalanx is not the human one's scaled copy (a flatter Wuji phalanx against a round human one), and position alone matches about a quarter of a robot finger's probes to the opposite face of the human finger (pad to nail), whose demonstrated distances then pull that face of the robot finger away from a grasped object. A weight of 0.5 (squared unit-normal differences lie in [0, 4], normalised squared distances are ~1) matches 96% of the probes to a like-facing source at unchanged coverage.

Functions:

normalize_cloud

normalize_cloud(points: ndarray) -> ndarray

Centre a point cloud on its centroid and scale it to unit root-mean-square radius.

PARAMETER	DESCRIPTION
points	(N, 3) points. TYPE: ndarray

RETURNS	DESCRIPTION
ndarray	(N, 3) normalised points.

Source code in eden/extensions/retargeting/otretarget/transport.py

def normalize_cloud(points: np.ndarray) -> np.ndarray:
    """Centre a point cloud on its centroid and scale it to unit root-mean-square radius.

    Parameters
    ----------
    points : np.ndarray
        ``(N, 3)`` points.

    Returns
    -------
    np.ndarray
        ``(N, 3)`` normalised points.
    """
    points = np.asarray(points, dtype=np.float64)
    centered = points - points.mean(axis=0, keepdims=True)
    rms = float(np.sqrt((centered**2).sum(axis=1).mean()))
    if not np.isfinite(rms) or rms <= 0.0:
        raise ValueError(
            f"normalize_cloud: a cloud of {len(points)} points has root-mean-square radius {rms!r}, so it cannot "
            f"be scaled to unit radius. Every point coincides: sample the surface with more than one distinct point."
        )
    return centered / rms

sinkhorn

sinkhorn(cost: ndarray, a: ndarray, b: ndarray, epsilon: float, max_iters: int = 2000, tol: float = 1e-09, device: str | device = 'cpu') -> ndarray

Solve min_P <P, C> + eps * sum P (log P - 1) over couplings with marginals a, b.

Log-domain Sinkhorn (Cuturi 2013), stable for small epsilon relative to the cost scale.

PARAMETER	DESCRIPTION
cost	(N, M) cost matrix C. TYPE: ndarray
a	(N,) source marginal, positive, summing to one. TYPE: ndarray
b	(M,) target marginal, positive, summing to one. TYPE: ndarray
epsilon	Entropic regularisation eps_ot (> 0). TYPE: float
max_iters	Iteration budget. TYPE: int DEFAULT: 2000
tol	Stop once the L1 violation of the source marginal falls below this. TYPE: float DEFAULT: 1e-09
device	Where the iterations run. The caller names it (the retargeter passes its env's device) rather than this function picking any available GPU, so a --cpu run stays off a shared GPU. TYPE: str | device DEFAULT: 'cpu'

RETURNS	DESCRIPTION
ndarray	(N, M) transport plan P*.

Source code in eden/extensions/retargeting/otretarget/transport.py

def sinkhorn(
    cost: np.ndarray,
    a: np.ndarray,
    b: np.ndarray,
    epsilon: float,
    max_iters: int = 2000,
    tol: float = 1e-9,
    device: str | torch.device = "cpu",
) -> np.ndarray:
    """Solve ``min_P <P, C> + eps * sum P (log P - 1)`` over couplings with marginals ``a``, ``b``.

    Log-domain Sinkhorn (Cuturi 2013), stable for small ``epsilon`` relative to the cost scale.

    Parameters
    ----------
    cost : np.ndarray
        ``(N, M)`` cost matrix ``C``.
    a : np.ndarray
        ``(N,)`` source marginal, positive, summing to one.
    b : np.ndarray
        ``(M,)`` target marginal, positive, summing to one.
    epsilon : float
        Entropic regularisation ``eps_ot`` (> 0).
    max_iters : int
        Iteration budget.
    tol : float
        Stop once the L1 violation of the source marginal falls below this.
    device : str | torch.device
        Where the iterations run. The caller names it (the retargeter passes its env's device) rather than this
        function picking any available GPU, so a ``--cpu`` run stays off a shared GPU.

    Returns
    -------
    np.ndarray
        ``(N, M)`` transport plan ``P*``.
    """
    if epsilon <= 0.0:
        raise ValueError(f"sinkhorn: epsilon={epsilon!r} must be > 0 (the paper uses eps_ot = 0.1).")
    scaled = -torch.as_tensor(np.asarray(cost), dtype=torch.float64, device=device) / epsilon
    log_a = torch.log(torch.as_tensor(np.asarray(a), dtype=torch.float64, device=device))
    log_b = torch.log(torch.as_tensor(np.asarray(b), dtype=torch.float64, device=device))
    # Scaled dual potentials u = f / eps, v = g / eps.
    u = torch.zeros_like(log_a)
    v = torch.zeros_like(log_b)
    for it in range(max_iters):
        u = log_a - torch.logsumexp(scaled + v[None, :], dim=1)
        v = log_b - torch.logsumexp(scaled + u[:, None], dim=0)
        # After the v-update the target marginal is exact; the source one measures convergence.
        if it % 10 == 9:
            row_mass = torch.exp(torch.logsumexp(scaled + u[:, None] + v[None, :], dim=1))
            if float((row_mass - torch.exp(log_a)).abs().sum()) < tol:
                break
    return torch.exp(scaled + u[:, None] + v[None, :]).cpu().numpy()

transport_correspondence

transport_correspondence(source: ndarray, target: ndarray, epsilon: float = 0.1, source_normals: ndarray | None = None, target_normals: ndarray | None = None, normal_weight: float = 0.0, source_weights: ndarray | None = None, target_weights: ndarray | None = None, device: str | device = 'cpu') -> ndarray

For each target point, the source point it receives the most mass from (src(p), Eq. 4).

Both clouds are normalised (:func:normalize_cloud) before the squared-Euclidean cost is built. The marginals are uniform over each cloud's points, or proportional to *_weights (the surface area a sample stands for) when a cloud is sampled at several densities: uniform marginals would hand the densely sampled part a larger share of the other surface.

PARAMETER	DESCRIPTION
source	(N, 3) source cloud (the human part, or the demonstrated object), in a common pose with target (the T-pose for human/robot, the object frame for object/object). TYPE: ndarray
target	(M, 3) target cloud (the robot link group, or the substitute object). TYPE: ndarray
epsilon	Entropic regularisation eps_ot. TYPE: float DEFAULT: 0.1
source_normals	(N, 3) / (M, 3) unit outward normals, required when normal_weight > 0. TYPE: ndarray DEFAULT: None
target_normals	(N, 3) / (M, 3) unit outward normals, required when normal_weight > 0. TYPE: ndarray DEFAULT: None
normal_weight	Weight of the normal-agreement cost |n_i - n_p|^2 (0: the paper's position-only cost). TYPE: float DEFAULT: 0.0
source_weights	(N,) / (M,) positive sample weights (normalised here); uniform when omitted. TYPE: ndarray DEFAULT: None
target_weights	(N,) / (M,) positive sample weights (normalised here); uniform when omitted. TYPE: ndarray DEFAULT: None
device	Where the Sinkhorn iterations run (see :func:sinkhorn). TYPE: str | device DEFAULT: 'cpu'

RETURNS	DESCRIPTION
ndarray	(M,) int indices into source.

Source code in eden/extensions/retargeting/otretarget/transport.py

def transport_correspondence(
    source: np.ndarray,
    target: np.ndarray,
    epsilon: float = 0.1,
    source_normals: np.ndarray | None = None,
    target_normals: np.ndarray | None = None,
    normal_weight: float = 0.0,
    source_weights: np.ndarray | None = None,
    target_weights: np.ndarray | None = None,
    device: str | torch.device = "cpu",
) -> np.ndarray:
    """For each target point, the source point it receives the most mass from (``src(p)``, Eq. 4).

    Both clouds are normalised (:func:`normalize_cloud`) before the squared-Euclidean cost is built.
    The marginals are uniform over each cloud's points, or proportional to ``*_weights`` (the surface area
    a sample stands for) when a cloud is sampled at several densities: uniform marginals would hand the
    densely sampled part a larger share of the other surface.

    Parameters
    ----------
    source : np.ndarray
        ``(N, 3)`` source cloud (the human part, or the demonstrated object), in a common pose with
        ``target`` (the T-pose for human/robot, the object frame for object/object).
    target : np.ndarray
        ``(M, 3)`` target cloud (the robot link group, or the substitute object).
    epsilon : float
        Entropic regularisation ``eps_ot``.
    source_normals, target_normals : np.ndarray, optional
        ``(N, 3)`` / ``(M, 3)`` unit outward normals, required when ``normal_weight`` > 0.
    normal_weight : float
        Weight of the normal-agreement cost ``|n_i - n_p|^2`` (0: the paper's position-only cost).
    source_weights, target_weights : np.ndarray, optional
        ``(N,)`` / ``(M,)`` positive sample weights (normalised here); uniform when omitted.
    device : str | torch.device
        Where the Sinkhorn iterations run (see :func:`sinkhorn`).

    Returns
    -------
    np.ndarray
        ``(M,)`` int indices into ``source``.
    """
    if len(source) == 0 or len(target) == 0:
        raise ValueError(
            f"transport_correspondence: cannot match a cloud of {len(source)} source points to one of {len(target)} "
            f"target points; both sides of a correspondence need surface samples."
        )
    src = normalize_cloud(source)
    tgt = normalize_cloud(target)
    cost = ((src[:, None, :] - tgt[None, :, :]) ** 2).sum(axis=-1)
    if normal_weight > 0.0:
        if source_normals is None or target_normals is None:
            raise ValueError(
                f"transport_correspondence: normal_weight={normal_weight!r} needs source_normals and target_normals; "
                f"pass both, or normal_weight=0 for the position-only cost."
            )
        diff = np.asarray(source_normals)[:, None, :] - np.asarray(target_normals)[None, :, :]
        cost = cost + normal_weight * (diff**2).sum(axis=-1)
    a = _marginal(source_weights, len(src))
    b = _marginal(target_weights, len(tgt))
    plan = sinkhorn(cost, a, b, epsilon, device=device)
    return plan.argmax(axis=0)

group_correspondences

group_correspondences(source_points: ndarray, source_labels: list[str], target_points: ndarray, target_labels: list[str], groups: list[tuple[tuple[str, ...], tuple[str, ...]]], epsilon: float = 0.1, source_normals: ndarray | None = None, target_normals: ndarray | None = None, normal_weight: float = 0.0, source_weights: ndarray | None = None, target_weights: ndarray | None = None, device: str | device = 'cpu') -> tuple[ndarray, ndarray]

Solve one transport plan per pre-assigned (source parts, target parts) group.

PARAMETER	DESCRIPTION
source_points	(N, 3) source samples (human probes in the rest pose). TYPE: ndarray
source_labels	N part names (the human joint each probe lies on). TYPE: list[str]
target_points	(M, 3) target samples (robot probes in the T-pose, same frame as the source). TYPE: ndarray
target_labels	M part names (the robot link each probe rides on). TYPE: list[str]
groups	((source parts...), (target parts...)); a target part may appear in one group only. TYPE: list[tuple[tuple[str, ...], tuple[str, ...]]]
epsilon	Entropic regularisation. TYPE: float DEFAULT: 0.1
source_normals	Unit outward normals of the samples (see :func:transport_correspondence). TYPE: ndarray DEFAULT: None
target_normals	Unit outward normals of the samples (see :func:transport_correspondence). TYPE: ndarray DEFAULT: None
normal_weight	Weight of the normal-agreement cost. TYPE: float DEFAULT: 0.0
source_weights	Sample weights (the area each sample stands for, see :func:transport_correspondence). TYPE: ndarray DEFAULT: None
target_weights	Sample weights (the area each sample stands for, see :func:transport_correspondence). TYPE: ndarray DEFAULT: None
device	Where the Sinkhorn iterations run (see :func:sinkhorn). TYPE: str | device DEFAULT: 'cpu'

RETURNS	DESCRIPTION
src	(M,) source index per target sample, -1 where the target's part is in no group. TYPE: ndarray
group	(M,) group index per target sample, -1 likewise. TYPE: ndarray

Source code in eden/extensions/retargeting/otretarget/transport.py

def group_correspondences(
    source_points: np.ndarray,
    source_labels: list[str],
    target_points: np.ndarray,
    target_labels: list[str],
    groups: list[tuple[tuple[str, ...], tuple[str, ...]]],
    epsilon: float = 0.1,
    source_normals: np.ndarray | None = None,
    target_normals: np.ndarray | None = None,
    normal_weight: float = 0.0,
    source_weights: np.ndarray | None = None,
    target_weights: np.ndarray | None = None,
    device: str | torch.device = "cpu",
) -> tuple[np.ndarray, np.ndarray]:
    """Solve one transport plan per pre-assigned (source parts, target parts) group.

    Parameters
    ----------
    source_points : np.ndarray
        ``(N, 3)`` source samples (human probes in the rest pose).
    source_labels : list[str]
        ``N`` part names (the human joint each probe lies on).
    target_points : np.ndarray
        ``(M, 3)`` target samples (robot probes in the T-pose, same frame as the source).
    target_labels : list[str]
        ``M`` part names (the robot link each probe rides on).
    groups : list[tuple[tuple[str, ...], tuple[str, ...]]]
        ``((source parts...), (target parts...))``; a target part may appear in one group only.
    epsilon : float
        Entropic regularisation.
    source_normals, target_normals : np.ndarray, optional
        Unit outward normals of the samples (see :func:`transport_correspondence`).
    normal_weight : float
        Weight of the normal-agreement cost.
    source_weights, target_weights : np.ndarray, optional
        Sample weights (the area each sample stands for, see :func:`transport_correspondence`).
    device : str | torch.device
        Where the Sinkhorn iterations run (see :func:`sinkhorn`).

    Returns
    -------
    src : np.ndarray
        ``(M,)`` source index per target sample, ``-1`` where the target's part is in no group.
    group : np.ndarray
        ``(M,)`` group index per target sample, ``-1`` likewise.
    """
    source_labels = np.asarray(source_labels)
    target_labels = np.asarray(target_labels)
    src = np.full(len(target_points), -1, dtype=np.int64)
    group_of = np.full(len(target_points), -1, dtype=np.int64)
    seen: dict[str, int] = {}
    for gi, (source_parts, target_parts) in enumerate(groups):
        for part in target_parts:
            if part in seen:
                raise ValueError(
                    f"group_correspondences: target part {part!r} is in groups {seen[part]} and {gi}; "
                    f"each robot link may be matched to one group of human parts only."
                )
            seen[part] = gi
        s_idx = np.flatnonzero(np.isin(source_labels, source_parts))
        t_idx = np.flatnonzero(np.isin(target_labels, target_parts))
        if len(s_idx) == 0 or len(t_idx) == 0:
            no_source = [p for p in source_parts if p not in set(source_labels.tolist())]
            no_target = [p for p in target_parts if p not in set(target_labels.tolist())]
            raise ValueError(
                f"group_correspondences: group {gi} {source_parts} <-> {target_parts} has {len(s_idx)} source and "
                f"{len(t_idx)} target samples; every group needs samples on both sides. Without samples: source "
                f"parts {no_source}, target parts {no_target} (a target link has none when it has no geometry or "
                f"probe_exclude_links drops it). Sampled source parts: {sorted(set(source_labels.tolist()))}; "
                f"sampled target parts: {sorted(set(target_labels.tolist()))}."
            )
        src[t_idx] = s_idx[
            transport_correspondence(
                source_points[s_idx],
                target_points[t_idx],
                epsilon,
                None if source_normals is None else source_normals[s_idx],
                None if target_normals is None else target_normals[t_idx],
                normal_weight,
                None if source_weights is None else source_weights[s_idx],
                None if target_weights is None else target_weights[t_idx],
                device,
            )
        ]
        group_of[t_idx] = gi
    return src, group_of

Capsule approximations of robot links, for the self-collision constraint (Sec. III-D).

Each link's surface is approximated by one capsule along its principal axis. A pair of links contributes a non-penetration row phi + n^T (J_a - J_b) dv >= -eps once their capsules come within an activation distance. Pairs that touch in the reference poses (adjacent links, a hand's palm and its finger roots) are excluded, as is usual for self-collision filtering.

Classes

LinkCapsules dataclass

LinkCapsules(link: ndarray, a: ndarray, b: ndarray, radius: ndarray, pairs: ndarray)

One capsule per link, in link frames.

ATTRIBUTE	DESCRIPTION
link	(C,) rig link index of each capsule. TYPE: ndarray
a, b	(C, 3) segment endpoints in the link frame. TYPE: ndarray
radius	(C,). TYPE: ndarray
pairs	(P, 2) capsule index pairs that are checked. TYPE: ndarray

Attributes

link instance-attribute

link: ndarray

a instance-attribute

a: ndarray

b instance-attribute

b: ndarray

radius instance-attribute

radius: ndarray

pairs instance-attribute

pairs: ndarray

Methods:

world

world(link_pos: ndarray, link_rot: ndarray) -> tuple[ndarray, ndarray]

World endpoints (C, 3), (C, 3).

Source code in eden/extensions/retargeting/otretarget/capsules.py

def world(self, link_pos: np.ndarray, link_rot: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    """World endpoints ``(C, 3)``, ``(C, 3)``."""
    rot = link_rot[self.link]
    pos = link_pos[self.link]
    return pos + np.einsum("cab,cb->ca", rot, self.a), pos + np.einsum("cab,cb->ca", rot, self.b)

Functions:

fit_capsule

fit_capsule(points: ndarray, radius_quantile: float) -> tuple[ndarray, ndarray, float]

Fit a capsule to surface points: principal axis, radius at a quantile of the distances to it.

PARAMETER	DESCRIPTION
points	(N, 3) surface points (link frame). TYPE: ndarray
radius_quantile	Quantile of the points' distance to the axis taken as the radius; below 1 the capsule sits inside protruding features rather than inflating around them. TYPE: float

RETURNS	DESCRIPTION
a, b : np.ndarray	Segment endpoints.
radius	TYPE: float

Source code in eden/extensions/retargeting/otretarget/capsules.py

def fit_capsule(points: np.ndarray, radius_quantile: float) -> tuple[np.ndarray, np.ndarray, float]:
    """Fit a capsule to surface points: principal axis, radius at a quantile of the distances to it.

    Parameters
    ----------
    points : np.ndarray
        ``(N, 3)`` surface points (link frame).
    radius_quantile : float
        Quantile of the points' distance to the axis taken as the radius; below 1 the capsule sits inside
        protruding features rather than inflating around them.

    Returns
    -------
    a, b : np.ndarray
        Segment endpoints.
    radius : float
    """
    centre = points.mean(axis=0)
    centred = points - centre
    _, _, vt = np.linalg.svd(centred, full_matrices=False)
    axis = vt[0]
    s = centred @ axis
    radial = np.linalg.norm(centred - s[:, None] * axis[None], axis=1)
    radius = float(np.quantile(radial, radius_quantile))
    lo, hi = float(s.min()) + radius, float(s.max()) - radius
    if lo > hi:
        lo = hi = 0.5 * (s.min() + s.max())
    return centre + lo * axis, centre + hi * axis, radius

build_link_capsules

build_link_capsules(link_points: dict[int, ndarray], link_parent: ndarray, reference_poses: list[tuple[ndarray, ndarray]], radius_quantile: float = 0.8, exclusion_margin: float = 0.005, link_names: list[str] | None = None, exclude: list[tuple[str, str]] | None = None) -> LinkCapsules

Fit capsules and choose the link pairs to check.

PARAMETER	DESCRIPTION
link_points	{rig link index: (N, 3) surface points in its frame} for links with geometry. TYPE: dict[int, ndarray]
link_parent	Rig link parent indices. TYPE: ndarray
reference_poses	Link poses (positions, rotations) at reference configurations; a pair closer than exclusion_margin in any of them is not checked. TYPE: list[tuple[ndarray, ndarray]]
radius_quantile	See :func:fit_capsule. TYPE: float DEFAULT: 0.8
exclusion_margin	Clearance below which a pair counts as touching in a reference pose. TYPE: float DEFAULT: 0.005
link_names	Rig link names (indexed like link_parent), needed by exclude. TYPE: list[str] DEFAULT: None
exclude	(regex_a, regex_b) link-name pairs never checked, in either order; for neighbouring links a capsule cannot separate (side-by-side fingers, whose capsules overlap as soon as they curl). TYPE: list[tuple[str, str]] DEFAULT: None

Source code in eden/extensions/retargeting/otretarget/capsules.py

def build_link_capsules(
    link_points: dict[int, np.ndarray],
    link_parent: np.ndarray,
    reference_poses: list[tuple[np.ndarray, np.ndarray]],
    radius_quantile: float = 0.8,
    exclusion_margin: float = 0.005,
    link_names: list[str] | None = None,
    exclude: list[tuple[str, str]] | None = None,
) -> LinkCapsules:
    """Fit capsules and choose the link pairs to check.

    Parameters
    ----------
    link_points : dict[int, np.ndarray]
        ``{rig link index: (N, 3) surface points in its frame}`` for links with geometry.
    link_parent : np.ndarray
        Rig link parent indices.
    reference_poses : list[tuple[np.ndarray, np.ndarray]]
        Link poses (positions, rotations) at reference configurations; a pair closer than
        ``exclusion_margin`` in any of them is not checked.
    radius_quantile : float
        See :func:`fit_capsule`.
    exclusion_margin : float
        Clearance below which a pair counts as touching in a reference pose.
    link_names : list[str], optional
        Rig link names (indexed like ``link_parent``), needed by ``exclude``.
    exclude : list[tuple[str, str]], optional
        ``(regex_a, regex_b)`` link-name pairs never checked, in either order; for neighbouring links a
        capsule cannot separate (side-by-side fingers, whose capsules overlap as soon as they curl).
    """
    links = sorted(link_points)
    a, b, radius = [], [], []
    for li in links:
        pa, pb, r = fit_capsule(link_points[li], radius_quantile)
        a.append(pa)
        b.append(pb)
        radius.append(r)
    caps = LinkCapsules(np.array(links), np.array(a), np.array(b), np.array(radius), np.zeros((0, 2), dtype=np.int64))
    ii, jj = np.triu_indices(len(links), k=1)
    li, lj = caps.link[ii], caps.link[jj]
    # Adjacent links (parent/child, siblings, grandparent) always touch at their joints.
    gp = np.where(link_parent >= 0, link_parent[np.maximum(link_parent, 0)], -1)
    adjacent = (
        (link_parent[li] == lj)
        | (link_parent[lj] == li)
        | ((link_parent[li] == link_parent[lj]) & (link_parent[li] >= 0))
        | (gp[li] == lj)
        | (gp[lj] == li)
    )
    keep = ~adjacent
    if exclude:
        if link_names is None:
            raise ValueError("build_link_capsules: exclude needs link_names to match its patterns against.")
        names = np.array(link_names)
        for pat_a, pat_b in exclude:
            a = np.array([re.fullmatch(pat_a, n) is not None for n in names])
            b = np.array([re.fullmatch(pat_b, n) is not None for n in names])
            keep &= ~((a[li] & b[lj]) | (b[li] & a[lj]))
    for pos, rot in reference_poses:
        wa, wb = caps.world(pos, rot)
        ca, cb = segment_closest_points(wa[ii], wb[ii], wa[jj], wb[jj])
        clearance = np.linalg.norm(ca - cb, axis=1) - caps.radius[ii] - caps.radius[jj]
        keep &= clearance > exclusion_margin
    caps.pairs = np.stack([ii[keep], jj[keep]], axis=1)
    return caps

The numerical core of OTRetarget's trust-region SQP: Gauss-Newton accumulation and the dense QP (ProxQP).

Nothing here knows about probes, channels or robots: residual rows arrive in factored form (:class:Rows) and leave as the Hessian, gradient and constraint matrices of one QP.

Classes

Rows dataclass

Rows(link: ndarray | None, u_r: ndarray | None, u_o: ndarray)

Residual rows in factored form: u_r^T J6[link] on the robot columns, u_o on the object columns.

Attributes

link instance-attribute

link: ndarray | None

u_r instance-attribute

u_r: ndarray | None

u_o instance-attribute

u_o: ndarray

Methods:

Local dataclass

Local(H: ndarray, g: ndarray, C: ndarray, l_in: ndarray, hard_lo: ndarray, hard_hi: ndarray, trust: ndarray, terms: dict[str, float], n_active: int, penetration: float, frozen: list, term_grads: dict[str, ndarray], contacts: list = list())

One linearisation of the frame problem: the QP data at a configuration and the true cost there.

Attributes

H instance-attribute

H: ndarray

g instance-attribute

g: ndarray

C instance-attribute

C: ndarray

l_in instance-attribute

l_in: ndarray

hard_lo instance-attribute

hard_lo: ndarray

hard_hi instance-attribute

hard_hi: ndarray

trust instance-attribute

trust: ndarray

terms instance-attribute

terms: dict[str, float]

n_active instance-attribute

n_active: int

penetration instance-attribute

penetration: float

frozen instance-attribute

frozen: list

term_grads instance-attribute

term_grads: dict[str, ndarray]

contacts class-attribute instance-attribute

contacts: list = field(default_factory=list)

cost property

cost: float

Methods:

GaussNewton

GaussNewton(n_x: int, n_v: int, link_jac: ndarray, terms)

Accumulates H = sum w J^T J and g = sum w J^T r for the QP, robot rows link by link.

terms names the cost terms (each reported with its value and gradient); add to another name is an error.

Source code in eden/extensions/retargeting/otretarget/solver.py

def __init__(self, n_x: int, n_v: int, link_jac: np.ndarray, terms) -> None:
    self.n_v = n_v
    self.link_jac = link_jac
    self.H = np.zeros((n_x, n_x))
    self.g = np.zeros(n_x)
    self.terms = dict.fromkeys(terms, 0.0)
    self.term_grads = {k: np.zeros(n_x) for k in terms}

Attributes

n_v instance-attribute

n_v = n_v

link_jac instance-attribute

link_jac = link_jac

H instance-attribute

H = np.zeros((n_x, n_x))

g instance-attribute

g = np.zeros(n_x)

terms instance-attribute

terms = dict.fromkeys(terms, 0.0)

term_grads instance-attribute

term_grads = {k: np.zeros(n_x) for k in terms}

Methods:

dense

dense(rows: Rows) -> ndarray

The rows as a dense (K, n_x) matrix (for constraints).

Source code in eden/extensions/retargeting/otretarget/solver.py

def dense(self, rows: Rows) -> np.ndarray:
    """The rows as a dense ``(K, n_x)`` matrix (for constraints)."""
    out = np.zeros((len(rows.u_o), len(self.g)))
    if rows.u_r is not None:
        out[:, : self.n_v] = np.einsum("ki,kix->kx", rows.u_r, self.link_jac[rows.link])
    out[:, self.n_v :] = rows.u_o
    return out

add

add(name: str, rows: Rows, r: ndarray, w: ndarray) -> None

Source code in eden/extensions/retargeting/otretarget/solver.py

def add(self, name: str, rows: Rows, r: np.ndarray, w: np.ndarray) -> None:
    if len(r) == 0:
        return
    n_v = self.n_v
    wr = w * r
    self.terms[name] += float((wr * r).sum())
    g_before = self.g.copy()
    if rows.u_o.shape[1]:
        self.H[n_v:, n_v:] += (rows.u_o * w[:, None]).T @ rows.u_o
        self.g[n_v:] += rows.u_o.T @ wr
    if rows.u_r is None:
        self.term_grads[name] += self.g - g_before
        return
    # Per-link sums of w u_r u_r^T, w r u_r and w u_r u_o^T, then one pass through each link Jacobian.
    links, inverse = np.unique(rows.link, return_inverse=True)
    order = np.argsort(inverse, kind="stable")
    bounds = np.flatnonzero(np.r_[True, np.diff(inverse[order]) != 0])
    u = rows.u_r[order]
    ws = w[order]
    m_rr = np.add.reduceat(ws[:, None, None] * u[:, :, None] * u[:, None, :], bounds, axis=0)
    g_r = np.add.reduceat(wr[order][:, None] * u, bounds, axis=0)
    J = self.link_jac[links]  # (L, 6, n_v)
    self.H[:n_v, :n_v] += np.einsum("lia,lij,ljb->ab", J, m_rr, J, optimize=True)
    self.g[:n_v] += np.einsum("lia,li->a", J, g_r)
    if rows.u_o.shape[1]:
        m_ro = np.add.reduceat(ws[:, None, None] * u[:, :, None] * rows.u_o[order][:, None, :], bounds, axis=0)
        cross = np.einsum("lia,lij->aj", J, m_ro)
        self.H[:n_v, n_v:] += cross
        self.H[n_v:, :n_v] += cross.T
    self.term_grads[name] += self.g - g_before

add_dense

add_dense(name: str, r: ndarray, J: ndarray, w: float) -> None

Source code in eden/extensions/retargeting/otretarget/solver.py

def add_dense(self, name: str, r: np.ndarray, J: np.ndarray, w: float) -> None:
    if len(r) == 0:
        return
    self.H += w * (J.T @ J)
    self.g += w * (J.T @ r)
    self.term_grads[name] += w * (J.T @ r)
    self.terms[name] += float(w * (r * r).sum())

Functions:

pad

pad(J: ndarray, n_x: int) -> ndarray

J widened with zero columns to n_x.

Source code in eden/extensions/retargeting/otretarget/solver.py

def pad(J: np.ndarray, n_x: int) -> np.ndarray:
    """``J`` widened with zero columns to ``n_x``."""
    out = np.zeros((len(J), n_x))
    out[:, : J.shape[1]] = J
    return out

single_threaded_blas

single_threaded_blas()

Context limiting the BLAS pools to one thread: faster for the solve's small products, and reproducible.

Source code in eden/extensions/retargeting/otretarget/solver.py

def single_threaded_blas():
    """Context limiting the BLAS pools to one thread: faster for the solve's small products, and reproducible."""
    try:
        from threadpoolctl import threadpool_limits
    except ImportError as exc:  # pragma: no cover - exercised only without the optional dependency
        raise ImportError(
            "OTRetargeter pins BLAS to one thread with threadpoolctl: install it with `pip install threadpoolctl` "
            "(the `ot-retargeter` extra of eden)."
        ) from exc
    return threadpool_limits(limits=1, user_api="blas")

elastic_shift

elastic_shift(C: ndarray, l_in: ndarray, lo: ndarray, hi: ndarray) -> ndarray | None

Smallest per-row relaxation t >= 0 (in the l1 sense) making C x >= l_in - t feasible in the box.

One LP: min sum t s.t. C x + t >= l_in, lo <= x <= hi, t >= 0. The l1 objective relaxes only the rows that actually conflict. Returns None if the LP solver fails.

Source code in eden/extensions/retargeting/otretarget/solver.py

def elastic_shift(C: np.ndarray, l_in: np.ndarray, lo: np.ndarray, hi: np.ndarray) -> np.ndarray | None:
    """Smallest per-row relaxation ``t >= 0`` (in the l1 sense) making ``C x >= l_in - t`` feasible in the box.

    One LP: ``min sum t  s.t.  C x + t >= l_in,  lo <= x <= hi,  t >= 0``. The l1 objective relaxes only
    the rows that actually conflict. Returns None if the LP solver fails.
    """
    from scipy.optimize import linprog
    from scipy.sparse import hstack as sparse_hstack
    from scipy.sparse import identity

    n, m = C.shape[1], C.shape[0]
    res = linprog(
        np.r_[np.zeros(n), np.ones(m)],
        A_ub=sparse_hstack([-C, -identity(m)], format="csr"),
        b_ub=-l_in,
        bounds=[*zip(lo, hi), *((0.0, None),) * m],
        method="highs",
    )
    return res.x[n:] if res.status == 0 else None

solve_qp

solve_qp(H, g, C, l_in, lo, hi) -> tuple[ndarray, str]

min 1/2 x^T H x + g^T x s.t. C x >= l_in, lo <= x <= hi with ProxQP (dense).

Linearised non-penetration rows can contradict each other (probes on both sides of a feature thinner than the distance grid), making the QP infeasible. When the plain solve does not converge, the SQP elastic relaxation is applied: :func:elastic_shift finds the smallest (l1) shift of the conflicting rows that restores feasibility, and the shifted QP is solved instead. If neither solve yields a finite step, the iterate does not move.

RETURNS	DESCRIPTION
x	The step, clipped to the box (zero when no solve succeeded). TYPE: ndarray
status	"solved", "relaxed" (solved after the elastic shift) or "failed". TYPE: str

Source code in eden/extensions/retargeting/otretarget/solver.py

def solve_qp(H, g, C, l_in, lo, hi) -> tuple[np.ndarray, str]:
    """``min 1/2 x^T H x + g^T x  s.t.  C x >= l_in,  lo <= x <= hi`` with ProxQP (dense).

    Linearised non-penetration rows can contradict each other (probes on both sides of a feature
    thinner than the distance grid), making the QP infeasible. When the plain solve does not converge,
    the SQP elastic relaxation is applied: :func:`elastic_shift` finds the smallest (l1) shift of the
    conflicting rows that restores feasibility, and the shifted QP is solved instead. If neither solve
    yields a finite step, the iterate does not move.

    Returns
    -------
    x : np.ndarray
        The step, clipped to the box (zero when no solve succeeded).
    status : str
        ``"solved"``, ``"relaxed"`` (solved after the elastic shift) or ``"failed"``.
    """
    try:
        from proxsuite import proxqp
    except ImportError as exc:  # pragma: no cover - exercised only without the optional dependency
        raise ImportError(
            "OTRetargeter solves its QPs with ProxQP: install it with `pip install proxsuite` "
            "(the `ot-retargeter` extra of eden)."
        ) from exc
    n = len(g)
    m = len(C)
    H = H + 1e-8 * np.eye(n)

    def attempt(lower: np.ndarray, max_iter: int, max_iter_in: int) -> np.ndarray | None:
        qp = proxqp.dense.QP(n, 0, m, True, proxqp.dense.HessianType.Dense)
        qp.settings.eps_abs = 1e-6  # 1 um on metre-scale steps
        qp.settings.max_iter = max_iter
        qp.settings.max_iter_in = max_iter_in
        qp.settings.verbose = False
        qp.init(H, g, None, None, C if m else None, lower if m else None, np.full(m, 1e20) if m else None, lo, hi)
        qp.solve()
        x = np.asarray(qp.results.x, dtype=np.float64)
        ok = qp.results.info.status == proxqp.QPSolverOutput.PROXQP_SOLVED and np.all(np.isfinite(x))
        return np.clip(x, lo, hi) if ok else None

    # Feasible iterates converge in a few dozen inner iterations; the tight cap makes an infeasible one
    # fail in milliseconds instead of exhausting a large budget (seconds) before the relaxation.
    x = attempt(l_in, 40, 20)
    if x is not None:
        return x, "solved"
    if m:
        shift = elastic_shift(C, l_in, lo, hi)
        if shift is not None:
            # A 10 um margin keeps the shifted problem off its feasibility boundary. A problem the LP finds feasible
            # as posed (no shift) was only slow: its solve counts as "solved", not "relaxed".
            relaxed = bool((shift > 1e-9).any())
            x = attempt(l_in - shift - 1e-5, 300, 50)
            if x is not None:
                return x, "relaxed" if relaxed else "solved"
            if not relaxed:
                x = attempt(l_in, 5000, 500)
                if x is not None:
                    return x, "solved"
    return np.zeros(n), "failed"

Interaction-fidelity metrics of a retargeted motion (OTRetarget Sec. IV-A).

Independently of the solver's probes and grids: every body unit (one correspondence group — a set of human parts and the robot links matched to them) is measured against every channel (the ground, each object) at every frame with exact signed distances from the posed human mesh vertices and the robot's visual-mesh vertices, each unit's distance being that of its closest vertex. Distances are to the channel's surface, MeshChannel.mesh: for a scan whose winding sign is unreliable that is its watertight solid, one object_solid_voxel outside the scan, for the demonstration and the robot alike.

With eps_eval = 20 mm, the demonstrated and retargeted interaction sets are H = {(t, u, c) : (d_hat)+ < eps} and R = {(t, u, c) : (d)+ < eps}; their agreement is reported as precision |H n R| / |R|, recall |H n R| / |H| and Jaccard |H n R| / |H u R|, and the geometric accuracy over H as the depth error mean |d - (d_hat)+| and the placement error mean |w - w_hat|. Ground and object interactions are reported separately, and the object interactions of the units on the welded hand entities (palm and phalanges: groups whose links all belong to a hand entity) once more on their own (hand/*), since a whole-body score hides a grasp that misses. A demonstrated witness on a substituted object (demo_mesh) is carried onto the scene mesh as the retargeter carries it before it is compared. Also reported: stance clearance and foot skate (the median over stance steps of the mean horizontal speed of the foot vertices within stance_height of the floor), the 95th-percentile scene penetration (and that of the hand units alone), the style orientation error, the object drift, and the smoothness of the motion as mean absolute second differences per frame of the joints, the root position and the human pelvis.

Classes

Functions:

evaluate_retarget

evaluate_retarget(retargeter: OTRetargeter, human: HumanSurfaceMotion, objects: list[InteractionObject], result: RetargetResult, eps_eval: float = 0.02, stance_height: float = 0.02, max_vertices_per_link: int = 1500) -> dict[str, float]

Score a retarget against its demonstration.

PARAMETER	DESCRIPTION
retargeter	The retargeter that produced result (after :meth:~OTRetargeter.retarget_motion, so its correspondence groups and channels match the demonstration). TYPE: OTRetargeter
human	The demonstration. TYPE: HumanSurfaceMotion
objects	Its objects (demonstrated poses). TYPE: list[InteractionObject]
result	The retargeted motion. TYPE: RetargetResult
eps_eval	Interaction threshold (20 mm). TYPE: float DEFAULT: 0.02
stance_height	A foot is in stance when its lowest point is below this (m). TYPE: float DEFAULT: 0.02
max_vertices_per_link	Robot visual-mesh vertices kept per link (evenly subsampled) to bound the cost. TYPE: int DEFAULT: 1500

RETURNS	DESCRIPTION
dict[str, float]	ground/*, object/* and hand/* agreement and errors, clearance_mm, skate_cm_s, penetration_p95_mm, hand/penetration_p95_mm, rot_deg (rot_body_deg over the root entity's links when hands are welded on), drift_cm, and jitter_joint_deg (deg/frame^2), jitter_root_cm and jitter_human_pelvis_cm (cm/frame^2).

Source code in eden/extensions/retargeting/otretarget/metrics.py

def evaluate_retarget(
    retargeter: OTRetargeter,
    human: HumanSurfaceMotion,
    objects: list[InteractionObject],
    result: RetargetResult,
    eps_eval: float = 0.02,
    stance_height: float = 0.02,
    max_vertices_per_link: int = 1500,
) -> dict[str, float]:
    """Score a retarget against its demonstration.

    Parameters
    ----------
    retargeter : OTRetargeter
        The retargeter that produced ``result`` (after :meth:`~OTRetargeter.retarget_motion`, so its
        correspondence groups and channels match the demonstration).
    human : HumanSurfaceMotion
        The demonstration.
    objects : list[InteractionObject]
        Its objects (demonstrated poses).
    result : RetargetResult
        The retargeted motion.
    eps_eval : float
        Interaction threshold (20 mm).
    stance_height : float
        A foot is in stance when its lowest point is below this (m).
    max_vertices_per_link : int
        Robot visual-mesh vertices kept per link (evenly subsampled) to bound the cost.

    Returns
    -------
    dict[str, float]
        ``ground/*``, ``object/*`` and ``hand/*`` agreement and errors, ``clearance_mm``, ``skate_cm_s``,
        ``penetration_p95_mm``, ``hand/penetration_p95_mm``, ``rot_deg`` (``rot_body_deg`` over the root entity's
        links when hands are welded on), ``drift_cm``, and
        ``jitter_joint_deg`` (deg/frame^2), ``jitter_root_cm`` and ``jitter_human_pelvis_cm`` (cm/frame^2).
    """
    kin = retargeter.kin
    links = {link.name: link for ent in kin.entities for link in ent.links}
    groups = [retargeter._group(g) for g in retargeter.correspondences]
    # Robot units: visual-mesh vertices of each group's links, in link frames.
    unit_link, unit_local, unit_id = [], [], []
    for ui, (_, robot_links) in enumerate(groups):
        for name in robot_links:
            link = links[name]
            if len(link.vgeoms) == 0:
                continue
            v = np.asarray(get_link_mesh(link, use_visual_mesh=True).vertices)
            if len(v) > max_vertices_per_link:
                v = v[np.linspace(0, len(v) - 1, max_vertices_per_link).astype(int)]
            unit_link.append(np.full(len(v), kin.link_index(name)))
            unit_local.append(v)
            unit_id.append(np.full(len(v), ui))
    hand_links = {link.name for ent in kin.entities[1:] for link in ent.links}
    # All of a group's links, not any: the Wuji presets put the hand's mount link in the elbow group.
    hand_unit = np.array([bool(robot_links) and set(robot_links) <= hand_links for _, robot_links in groups])
    unit_link = np.concatenate(unit_link)
    unit_local = np.concatenate(unit_local)
    unit_id = np.concatenate(unit_id)
    # Human units: mesh vertices of each group's parts.
    vparts = human.vertex_parts
    human_unit = np.full(len(vparts), -1)
    for ui, (human_parts, _) in enumerate(groups):
        human_unit[np.isin(vparts, [human.joint_index(p) for p in human_parts])] = ui

    n_frames, n_units = len(result.qpos), len(groups)
    # Each unit's vertex indices, found once: rebuilding the masks per frame, channel and unit cost O(V * units).
    human_idx = [np.flatnonzero(human_unit == ui) for ui in range(n_units)]
    robot_idx = [np.flatnonzero(unit_id == ui) for ui in range(n_units)]
    channels = retargeter.channels
    d_hat = np.full((n_frames, n_units, len(channels)), np.inf)
    d_rob = np.full_like(d_hat, np.inf)
    w_hat = np.zeros((*d_hat.shape, 3))
    w_rob = np.zeros_like(w_hat)
    pen = np.zeros(n_frames)
    hand_pen = np.zeros(n_frames)
    hand_vertex = hand_unit[unit_id]
    foot_units = [ui for ui, (hp, _) in enumerate(groups) if any("ankle" in p or "Foot" in p for p in hp)]
    foot_low = np.full((n_frames, len(foot_units)), np.inf)
    foot_pts: list[list[np.ndarray]] = [[] for _ in foot_units]
    solved = {o.name: result.object_poses[o.name] for o in objects}
    rot_err = []
    for t in range(n_frames):
        kin.set_configuration(result.qpos[t])
        link_pos, link_rot = kin.link_poses()
        robot_world = link_pos[unit_link] + np.einsum("nab,nb->na", link_rot[unit_link], unit_local)
        human_world = human.vertices[t].astype(np.float64)
        for ci, channel in enumerate(channels):
            oi = retargeter._channel_object[ci]
            if oi < 0:
                R_demo = R_now = np.eye(3)
                tau_demo = tau_now = np.array([0.0, 0.0, retargeter.ground_height])
            else:
                R_demo = quat_to_R(objects[oi].poses[t, 3:])
                tau_demo = objects[oi].poses[t, :3]
                R_now = quat_to_R(solved[objects[oi].name][t, 3:])
                tau_now = solved[objects[oi].name][t, :3]
            demo_channel = retargeter._demo_channels.get(oi, channel)
            hd, hw = demo_channel.exact_distance((human_world - tau_demo) @ R_demo)
            rd, rw = channel.exact_distance((robot_world - tau_now) @ R_now)
            pen[t] = max(pen[t], float(np.clip(-rd, 0.0, None).max()))
            if oi >= 0 and hand_vertex.any():
                hand_pen[t] = max(hand_pen[t], float(np.clip(-rd[hand_vertex], 0.0, None).max()))
            for ui in range(n_units):
                if len(hi := human_idx[ui]):
                    k = hi[np.argmin(hd[hi])]
                    d_hat[t, ui, ci], w_hat[t, ui, ci] = hd[k], hw[k]
                if len(ri := robot_idx[ui]):
                    k = ri[np.argmin(rd[ri])]
                    d_rob[t, ui, ci], w_rob[t, ui, ci] = rd[k], rw[k]
            if oi in retargeter._witness_map:
                # The demonstrated witness lies on the demo mesh; the robot's on the scene mesh. Mapped after the
                # closest vertex is chosen, so only one witness per unit goes through the tree.
                tree, s_pts, _ = retargeter._witness_map[oi]
                w_hat[t, :, ci] = s_pts[tree.query(w_hat[t, :, ci])[1]]
        for k, ui in enumerate(foot_units):
            pts = robot_world[robot_idx[ui]]
            foot_low[t, k] = pts[:, 2].min() - retargeter.ground_height
            foot_pts[k].append(pts)
        # Style orientation error against the retargeter's own targets.
        _, style_rot = retargeter._style_targets(human, t)
        rel = np.einsum("lab,lcb->lac", link_rot[retargeter._style_link_idx], style_rot)
        cos = np.clip((np.trace(rel, axis1=1, axis2=2) - 1.0) / 2.0, -1.0, 1.0)
        rot_err.append(np.degrees(np.arccos(cos)))

    out: dict[str, float] = {}
    is_ground = np.array([retargeter._channel_object[ci] < 0 for ci in range(len(channels))])
    every_unit = np.ones(n_units, dtype=bool)
    for name, mask, units in (
        ("ground", is_ground, every_unit),
        ("object", ~is_ground, every_unit),
        ("hand", ~is_ground, hand_unit),
    ):
        if not mask.any() or not units.any():
            continue
        dh = d_hat[:, units][:, :, mask]
        dr = d_rob[:, units][:, :, mask]
        H = np.maximum(dh, 0.0) < eps_eval
        R = np.maximum(dr, 0.0) < eps_eval
        out |= {f"{name}/{k}": v for k, v in _agreement(H, R).items()}
        if H.any():
            out[f"{name}/depth_err_mm"] = 1e3 * float(np.abs(dr[H] - np.maximum(dh[H], 0.0)).mean())
            out[f"{name}/place_err_mm"] = 1e3 * float(
                np.linalg.norm(w_rob[:, units][:, :, mask][H] - w_hat[:, units][:, :, mask][H], axis=1).mean()
            )
    stance = foot_low < stance_height
    out["clearance_mm"] = 1e3 * float(np.median(foot_low[stance])) if stance.any() else float("nan")
    skate = []
    dt = 1.0 / result.fps
    for k in range(len(foot_units)):
        for t in range(n_frames - 1):
            if stance[t, k] and stance[t + 1, k]:
                # The vertices in contact at t: the least moving vertex of the whole foot read a foot yawing or
                # rolling heel-to-toe in place as no skate (any rigid motion has a point that barely moves).
                pts = foot_pts[k][t]
                down = pts[:, 2] - retargeter.ground_height < stance_height
                disp = np.linalg.norm((foot_pts[k][t + 1][down] - pts[down])[:, :2], axis=1)
                skate.append(disp.mean() / dt)
    out["skate_cm_s"] = 100.0 * float(np.median(skate)) if skate else float("nan")
    out["penetration_p95_mm"] = 1e3 * float(np.percentile(pen, 95))
    if hand_unit.any() and (~is_ground).any():
        out["hand/penetration_p95_mm"] = 1e3 * float(np.percentile(hand_pen, 95))
    if n_frames > 2:
        q = result.qpos
        out["jitter_joint_deg"] = float(np.degrees(np.abs(np.diff(q[:, kin.n_base_q :], 2, axis=0))).mean())
        if kin.floating:
            out["jitter_root_cm"] = 100.0 * float(np.linalg.norm(np.diff(q[:, :3], 2, axis=0), axis=1).mean())
            pelvis = human.joint_pos[:n_frames, retargeter._pelvis_joint]
            out["jitter_human_pelvis_cm"] = 100.0 * float(np.linalg.norm(np.diff(pelvis, 2, axis=0), axis=1).mean())
    rot_err = np.array(rot_err)  # (T, tracked links)
    out["rot_deg"] = float(rot_err.mean())
    # The same over the links of the root entity alone: the hand presets weigh finger style down on purpose.
    body = np.array([kin.link_names[li] not in hand_links for li in retargeter._style_link_idx])
    if body.any() and not body.all():
        out["rot_body_deg"] = float(rot_err[:, body].mean())
    if objects:
        out["drift_cm"] = 100.0 * float(
            np.mean([np.linalg.norm(solved[o.name][:, :3] - o.poses[:n_frames, :3], axis=1).mean() for o in objects])
        )
    return out

SomaRetargeter

SomaRetargeter runs the eden-soma-retargeter library in a worker process. That extra pins a trimesh that only a checkout's uv override reconciles with Genesis, so install it with uv sync --inexact --extra soma-retargeter (it cannot be pip-installed).

Eden RetargetingBase wrapper around the eden-soma-retargeter library.

Retargets human motion — SMPL / SMPL-H / SMPL-X (fitted via py-soma-x) or native SOMA (arrays / save_soma_npz files / SOMA-skeleton BVH) — onto a floating-base robot (Unitree G1 29-DOF, LimX Oli 31-DOF), returning Genesis-compatible qpos.

Usage mirrors DexRetargeter / OmniRetargeter:

.. code-block:: python

retargeter = SomaRetargeter(
    env=env,
    options=SomaRetargeter.configure(
        entity_name="robot",
        body_model="smplx",
        body_model_path="/path/to/SMPLX_NEUTRAL.npz",
        target_fps=30.0,
    ),
)
qpos = retargeter.retarget_motion(poses=..., trans=..., betas=..., source_fps=120.0)
qpos_batch = retargeter.retarget_motions([clip0, clip1, ...])

Why a worker subprocess. Retargeting runs in a persistent Genesis-free worker process; this class streams clips to it and permutes results into entity.dofs_name order. The isolation is precautionary — it keeps the Warp fit's CUDA context and its several GB off the ones Genesis holds — and is an internal detail: the public API is the same in-process loop as the other retargeters.

Classes

SomaRetargeter

SomaRetargeter(env: 'EnvBase', options: RetargeterOptions)

Bases: RetargetingBase

SMPL/SMPL-H/SMPL-X/SOMA -> robot retargeter (library-backed).

PARAMETER	DESCRIPTION
body_model	Source motion representation. The smpl family is fitted to the SOMA rig via py-soma-x; "soma" consumes SOMA motion directly. TYPE: ('smpl', 'smplh', 'smplx', 'soma') DEFAULT: "smpl"
body_model_path	Path to the body-model file (.npz/.pkl) or a directory holding per-gender <MODEL>_<GENDER>.{npz,pkl} files. Required for the smpl family; unused for "soma". TYPE: str
gender	Default gender; a per-call gender= on retarget_motion overrides it. TYPE: ('neutral', 'male', 'female') DEFAULT: "neutral"
target_fps	FPS every clip is resampled to before retargeting. TYPE: float
snap_to_ground	Post-process each clip with one constant vertical offset so the lowest point of the robot's collision geometry over the whole motion touches the ground plane (z = 0) — lifts clips whose feet sink below the floor, lowers floating ones. Sweeps the clip through the entity's forward kinematics; the entity's qpos is restored afterwards (velocities zeroed). TYPE: bool
device	Torch/warp device for the worker ("cuda", "cuda:1", "cpu"). TYPE: str
enable_procedural_transforms	Fit through py-soma-x's expanded twist-joint rig instead of the public 78-joint SOMA rig. Off by default (the library defaults it on) because it requires SOMA_procedural_transforms.json / SOMA_template_rig.usda, which the public nvidia/soma-x asset repo does not ship — turning it on without them raises FileNotFoundError naming the missing file. Both modes emit the same 78-joint interface, so the mode never changes what callers do; the fitted rotations do differ, since procedural mode refines the skin the fit is matched against. Ignored by body_model="soma" (no fit). TYPE: bool
robot	Library identifier of the target robot ("unitree_g1", "limx_oli", or any robot the library ships a configs/<robot>/ bundle for). Must match the entity referenced by entity_name: the library emits that robot's joint names, which are permuted into entity.dofs_name by name. For robots whose library config bundles no downloadable model (e.g. "limx_oli"), the entity's own resolved model file is forwarded as robot_model_path, so both sides use the same asset. TYPE: str
retargeter_kwargs	Extra keyword arguments forwarded verbatim to the library MotionRetargeter (e.g. fit_chunk, forward_chunk, ik_config_overrides, fit_kwargs). A robot_model_path here overrides the entity-derived one. TYPE: dict

Source code in eden/extensions/retargeting/soma_retargeter/retargeter.py

def __init__(self, env: "EnvBase", options: RetargeterOptions) -> None:
    # Set before anything that can raise, so __del__ -> close() is always safe.
    self._proc: subprocess.Popen | None = None
    self._perm: np.ndarray | None = None
    self._perm_names: tuple[str, ...] | None = None

    super().__init__(env, options)

    if self.entity.has_immobile_base:
        reason = "is welded onto another entity" if self.entity.is_attached else "is fixed-base"
        raise ValueError(
            f"SomaRetargeter requires a floating-base entity (returns qpos with "
            f"[base_pos(3), base_quat(4), joints(N)]); '{self.entity.name}' {reason} and has "
            f"no base DOFs."
        )
    check_backend_available(self.body_model)

    # Built (and validated) before the worker exists, so a bad option costs no process.
    self._start_worker(self._init_message())

Attributes

body_model class-attribute instance-attribute

body_model: Literal['smpl', 'smplh', 'smplx', 'soma'] = 'smplx'

body_model_path class-attribute instance-attribute

body_model_path: str = ''

gender class-attribute instance-attribute

gender: Literal['neutral', 'male', 'female'] = 'neutral'

target_fps class-attribute instance-attribute

target_fps: float = 30.0

snap_to_ground class-attribute instance-attribute

snap_to_ground: bool = False

device class-attribute instance-attribute

device: str = 'cuda'

enable_procedural_transforms class-attribute instance-attribute

enable_procedural_transforms: bool = False

robot class-attribute instance-attribute

robot: str = 'unitree_g1'

retargeter_kwargs class-attribute instance-attribute

retargeter_kwargs: dict = {}

joint_names property

joint_names: list[str]

Full DOF name list of the underlying entity (dofs_name order).

Methods:

close

close() -> None

Terminate the worker subprocess (idempotent).

Source code in eden/extensions/retargeting/soma_retargeter/retargeter.py

def close(self) -> None:
    """Terminate the worker subprocess (idempotent)."""
    proc, self._proc = getattr(self, "_proc", None), None
    if proc is None:
        return
    try:
        if proc.stdin is not None:
            proc.stdin.close()
        proc.wait(timeout=10)
    except Exception:  # noqa: BLE001 - best-effort teardown
        proc.kill()

retarget_motion

retarget_motion(poses: ndarray | None = None, trans: ndarray | None = None, transl: ndarray | None = None, betas: ndarray | None = None, gender: str | None = None, source_fps: float | None = None, **extra) -> ndarray

Retarget one clip to Genesis qpos (T, 7 + N) in entity.dofs_name order.

Field names follow SOMA-X conventions. For the smpl family pass poses / trans / betas / source_fps (+ optional gender); for native SOMA arrays pass poses / transl / source_fps. Returned qpos is [x, y, z, qw, qx, qy, qz, joints...], Z-up, wxyz quaternion, radians.

Source code in eden/extensions/retargeting/soma_retargeter/retargeter.py

def retarget_motion(
    self,
    poses: np.ndarray | None = None,
    trans: np.ndarray | None = None,
    transl: np.ndarray | None = None,
    betas: np.ndarray | None = None,
    gender: str | None = None,
    source_fps: float | None = None,
    **extra,
) -> np.ndarray:
    """Retarget one clip to Genesis qpos ``(T, 7 + N)`` in ``entity.dofs_name`` order.

    Field names follow SOMA-X conventions. For the smpl family pass
    ``poses`` / ``trans`` / ``betas`` / ``source_fps`` (+ optional
    ``gender``); for native SOMA arrays pass ``poses`` / ``transl`` /
    ``source_fps``. Returned qpos is ``[x, y, z, qw, qx, qy, qz, joints...]``,
    Z-up, wxyz quaternion, radians.
    """
    clip: dict = {"source_fps": source_fps, **extra}
    if poses is not None:
        clip["poses"] = poses
    if trans is not None:
        clip["trans"] = trans
    if transl is not None:
        clip["transl"] = transl
    if betas is not None:
        clip["betas"] = betas
    if gender is not None:
        clip["gender"] = gender
    clip = {k: v for k, v in clip.items() if v is not None}
    return self._to_qpos(self._run([clip])[0])

retarget_bvh

retarget_bvh(bvh_path) -> ndarray

Retarget a SOMA-skeleton BVH file (e.g. BONES-SEED) to qpos.

Source code in eden/extensions/retargeting/soma_retargeter/retargeter.py

def retarget_bvh(self, bvh_path) -> np.ndarray:
    """Retarget a SOMA-skeleton BVH file (e.g. BONES-SEED) to qpos."""
    return self._to_qpos(self._run([{"bvh_path": str(bvh_path)}])[0])

retarget_motions

retarget_motions(clips: list[dict]) -> list[ndarray]

Retarget many clips in one batched IK solve (one env per clip).

Each clip is the keyword dict of a single-clip call ({"poses", "trans", "source_fps", ...} / {"poses", "transl", ...} / {"bvh_path": ...}).

Source code in eden/extensions/retargeting/soma_retargeter/retargeter.py

def retarget_motions(self, clips: list[dict]) -> list[np.ndarray]:
    """Retarget many clips in one batched IK solve (one env per clip).

    Each clip is the keyword dict of a single-clip call
    (``{"poses", "trans", "source_fps", ...}`` / ``{"poses", "transl", ...}``
    / ``{"bvh_path": ...}``).
    """
    return [self._to_qpos(r) for r in self._run(clips)]

Functions:

build_dof_permutation

build_dof_permutation(source_joint_names: Sequence[str], entity_dofs_name: Sequence[str]) -> ndarray

Permutation perm such that out[:, perm] = src maps each source DOF to its entity slot.

Resolves purely by name (independent of Genesis's internal link/DOF order, which changed BFS->DFS in genesis 1.2.0). A trailing _dof suffix on source names is tolerated. Raises ValueError on a length mismatch or a missing joint, listing both name sets.

Source code in eden/extensions/retargeting/soma_retargeter/retargeter.py

def build_dof_permutation(source_joint_names: Sequence[str], entity_dofs_name: Sequence[str]) -> np.ndarray:
    """Permutation ``perm`` such that ``out[:, perm] = src`` maps each source DOF to its entity slot.

    Resolves purely by name (independent of Genesis's internal link/DOF order,
    which changed BFS->DFS in genesis 1.2.0). A trailing ``_dof`` suffix on
    source names is tolerated. Raises ``ValueError`` on a length mismatch or a
    missing joint, listing both name sets.
    """
    entity = list(entity_dofs_name)
    entity_index = {n: i for i, n in enumerate(entity)}
    # Strip the CSV '_dof' suffix only when doing so resolves a name.
    src = [
        n if n in entity_index else (n[:-4] if n.endswith("_dof") and n[:-4] in entity_index else n)
        for n in source_joint_names
    ]
    if len(src) != len(entity):
        raise ValueError(
            f"Retargeter emits {len(src)} DOFs but entity has {len(entity)}. Source: {src}. Entity: {entity}."
        )
    missing = [n for n in src if n not in entity_index]
    if missing:
        raise ValueError(f"Retargeter DOFs not in entity.dofs_name: {missing}. Entity DOFs: {entity}.")
    perm = [entity_index[n] for n in src]
    if len(set(perm)) != len(perm):
        raise ValueError(f"Retargeter DOF names are not a bijection onto entity.dofs_name (duplicates). Source: {src}.")
    return np.array(perm, dtype=np.int64)

FlashDexRetarget

The kinematic half of FlashDexRetarget: import a released hand-object dataset, preprocess and filter its clips, retarget each with physics IK in MuJoCo, and pack the kept clips into motion.pt.

Options for the FlashDexRetarget kinematic-retargeting pipeline (human demos -> motion.pt).

Two stages, each driven by one options object:

• :class:FDRImportOptions — a released dataset (TACO, OakInk2, HOT3D, DexYCB, each read by its :mod:eden.datasets loader) -> <dataset>/human_demo/<clip>.npz and <dataset>/objects/<id>/visual.obj.
• :class:FDRKinematicRetargetingOptions — those clips -> support disks and reject rules (report.csv), convex object parts, MuJoCo physics IK onto the robot hand, and <dataset>/<robot>/motion.pt.

Defaults are FlashDexRetarget's (Holiday-Robot/FlashDexRetarget config/retarget/*.yaml); the Isaac-only usd step is gone. eden retarget flashdex builds these from --config / --override the way eden train does.

Attributes

FDRSourceOptions module-attribute

FDRSourceOptions = Annotated[Annotated[FDRTacoSourceOptions, Tag('taco')] | Annotated[FDROakInk2SourceOptions, Tag('oakink2')] | Annotated[FDRHot3dSourceOptions, Tag('hot3d')] | Annotated[FDRDexYCBSourceOptions, Tag('dexycb')], _by_name('source', ('taco', 'oakink2', 'hot3d', 'dexycb'), None)]

FDRConvexOptions module-attribute

FDRConvexOptions = Annotated[Annotated[FDRCoacdConvexOptions, Tag('coacd')] | Annotated[FDRCuacdConvexOptions, Tag('cuacd')], _by_name('convex', ('coacd', 'cuacd'), 'coacd')]

Classes

FDRTacoSourceOptions

FDRTacoSourceOptions(**data)

Bases: ConfigurableOptions

TACO, read by :class:~eden.datasets.TacoDataset (30 Hz, one clip per recording).

PARAMETER	DESCRIPTION
table_z	Height to put the surface the objects rest on at; None keeps the release's heights. TYPE: float | None
dataset	Options of :class:~eden.datasets.TacoDataset; checked when it is built. raw_data_path, body_model_dir, body_model_params, target_framerate and sides come from the import options. TYPE: dict[str, Any]

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

name class-attribute instance-attribute

name: Literal['taco'] = 'taco'

table_z class-attribute instance-attribute

table_z: float | None = None

dataset class-attribute instance-attribute

dataset: dict[str, Any] = Field(default_factory=dict)

FDROakInk2SourceOptions

FDROakInk2SourceOptions(**data)

Bases: ConfigurableOptions

OakInk2, read by :class:~eden.datasets.OakInk2Dataset (120 Hz, one clip per program stage).

PARAMETER	DESCRIPTION
table_z	Height to put the surface the objects rest on at; None keeps the release's heights. TYPE: float | None
desk_over_recording	Fit the support disks on every object of the whole recording (CHORD), not only the clip's own. TYPE: bool
scales	Mesh scale per object id (1.0 when absent): OakInk2's scans of these are off in size. TYPE: dict[str, float]
dataset	Options of :class:~eden.datasets.OakInk2Dataset (e.g. contact, the distance at which a MANO vertex touches an object); checked when it is built. raw_data_path, body_model_dir, body_model_params, target_framerate and sides come from the import options. TYPE: dict[str, Any]

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

name class-attribute instance-attribute

name: Literal['oakink2'] = 'oakink2'

table_z class-attribute instance-attribute

table_z: float | None = None

desk_over_recording class-attribute instance-attribute

desk_over_recording: bool = True

scales class-attribute instance-attribute

scales: dict[str, float] = Field(default_factory=lambda: {'O02@0015@00001': 1.1, 'O02@0015@00004': 1.08, 'O02@0015@00007': 1.2, 'O02@0015@00010': 1.2, 'O02@0015@00013': 1.2, 'O02@0015@00019': 1.02, 'O02@0015@00020': 1.05, 'O02@0015@00022': 1.2, 'O02@0015@00025': 1.2, 'O02@0018@00001': 1.05, 'O02@0029@00002': 1.3, 'O02@0029@00009': 1.3, 'O02@0029@00011': 0.9, 'O02@0029@00012': 1.3, 'O02@0206@00001': 1.15, 'O02@0206@00002': 0.95})

dataset class-attribute instance-attribute

dataset: dict[str, Any] = Field(default_factory=dict)

FDRHot3dSourceOptions

FDRHot3dSourceOptions(**data)

Bases: ConfigurableOptions

HOT3D, read by :class:~eden.datasets.Hot3dDataset (30 Hz, cut into hand-object segments).

PARAMETER	DESCRIPTION
table_z	Height to put the surface the objects rest on at; None keeps the release's heights. TYPE: float | None
desk_over_recording	Fit the support disks on every object of the whole recording, not only the segment's own. TYPE: bool
dataset	Options of :class:~eden.datasets.Hot3dDataset, e.g. its segmenting (tip_contact, link_contact, gap_frames, min_frames, pad_frames, modes); checked when it is built. raw_data_path, body_model_dir, body_model_params, target_framerate and sides come from the import options. TYPE: dict[str, Any]

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

name class-attribute instance-attribute

name: Literal['hot3d'] = 'hot3d'

table_z class-attribute instance-attribute

table_z: float | None = 1.0

desk_over_recording class-attribute instance-attribute

desk_over_recording: bool = False

dataset class-attribute instance-attribute

dataset: dict[str, Any] = Field(default_factory=dict)

FDRDexYCBSourceOptions

FDRDexYCBSourceOptions(**data)

Bases: ConfigurableOptions

DexYCB, read by :class:~eden.datasets.DexYCBDataset (30 Hz, one hand per clip).

Not one of FlashDexRetarget's releases: a single-hand grasp-and-lift source, so import it with sides set to the hand the subject used (every subject's mano_sides is right in the public release). The release's root holds <subject>/<sequence>/, calibration/ and models/; poses stay in the AprilTag frame, on the release's meshes.

PARAMETER	DESCRIPTION
table_z	Height to put the surface the objects rest on at; None keeps the AprilTag frame's (the table top is at about 0). TYPE: float | None
dataset	Options of :class:~eden.datasets.DexYCBDataset; checked when it is built. raw_data_path, body_model_dir, body_model_params, target_framerate, calibration_dir, models_dir, ground_initial_pose and apply_mesh_correction come from the import. TYPE: dict[str, Any]

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

name class-attribute instance-attribute

name: Literal['dexycb'] = 'dexycb'

table_z class-attribute instance-attribute

table_z: float | None = None

dataset class-attribute instance-attribute

dataset: dict[str, Any] = Field(default_factory=dict)

FDRImportOptions

FDRImportOptions(**data)

Bases: ConfigurableOptions

A released dataset -> <dataset>/human_demo/<clip>.npz + <dataset>/objects/<id>/visual.obj.

PARAMETER	DESCRIPTION
source	The release and its loader settings, picked by name (taco / oakink2 / hot3d / dexycb). TYPE: FDRSourceOptions
raw	The downloaded release's root directory. TYPE: str
dataset	The dataset directory to write. TYPE: str
mano_dir	Directory holding MANO_RIGHT.pkl and MANO_LEFT.pkl (mano.is.tue.mpg.de; not redistributable). TYPE: str
sides	Hands to import; a clip without an object in each of them is skipped. DexYCB has one hand per clip, so it needs right or left. TYPE: Literal['both', 'right', 'left']
fps	Rate every clip is resampled to. TYPE: float
max_obj_rot_step_deg	A clip whose object turns more than this in one source frame is skipped (a pose flip). TYPE: float
textures	Export each object with the release's materials and textures, for rendering visual.obj. Off, it is exported from its geometry alone (a mesh with UVs gets trimesh's grey placeholder texture), which is all the retargeting reads; an object whose geometry is unchanged keeps the materials an earlier import wrote. On, every import loads each object's textures once more (a 4096^2 one decodes to ~50 MB, one object at a time) to compare them with what it wrote last time, and re-encodes them where they changed. TYPE: bool

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

source instance-attribute

source: FDRSourceOptions

raw instance-attribute

raw: str

dataset instance-attribute

dataset: str

mano_dir instance-attribute

mano_dir: str

sides class-attribute instance-attribute

sides: Literal['both', 'right', 'left'] = 'both'

fps class-attribute instance-attribute

fps: float = Field(60.0, gt=0)

max_obj_rot_step_deg class-attribute instance-attribute

max_obj_rot_step_deg: float = Field(60.0, gt=0)

textures class-attribute instance-attribute

textures: bool = False

FDRStepsOptions

FDRStepsOptions(**data)

Bases: ConfigurableOptions

Which steps of the kinematic pipeline run (preprocess always does, when the dataset has clips).

PARAMETER	DESCRIPTION
convex	Convex decomposition of every object (<object>/convex/<i>.obj). TYPE: bool
retarget	Physics IK of every kept clip (<robot>/retargeted/<clip>.npz). TYPE: bool
pack	Kept clips -> <robot>/motion.pt. TYPE: bool
urdf	Object URDFs for simulation (<object>/object.urdf); needs <robot>/motion.pt. TYPE: bool
sdf	Object SDF grids (<object>/.sdf_cache/); needs <robot>/motion.pt. TYPE: bool

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

convex class-attribute instance-attribute

convex: bool = True

retarget class-attribute instance-attribute

retarget: bool = True

pack class-attribute instance-attribute

pack: bool = True

urdf class-attribute instance-attribute

urdf: bool = True

sdf class-attribute instance-attribute

sdf: bool = True

FDRDeskOptions

FDRDeskOptions(**data)

Bases: ConfigurableOptions

Support disks (tables, shelves) fitted where the objects rest still; ported from CHORD.

PARAMETER	DESCRIPTION
mode	fit the disks, keep a clip's own support_disks (fit when absent), none: the floor only. TYPE: Literal['fit', 'keep', 'none']
still_speed	Linear speed (m/s) under which an object rests. TYPE: float
still_ang_speed	Angular speed (rad/s) under which an object rests. TYPE: float
min_still_sec	Shortest rest that fits a disk. TYPE: float
disk_height	Height of a fitted disk. TYPE: float
phantom_in_disk_frac	A disk that another simulated object occupies this fraction of the time is that object, dropped. TYPE: float
phantom_z	Vertical band (m) of that occupancy test. TYPE: float
consolidate_z	Overlapping disks within this height difference merge. TYPE: float
rest_band	An object passes over a disk within [lo, hi] of its top: the disk is used. TYPE: tuple[float, float]
contact_z	With the recording's context: an object stands on a disk within this distance of its top. TYPE: float

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

mode class-attribute instance-attribute

mode: Literal['fit', 'keep', 'none'] = 'fit'

still_speed class-attribute instance-attribute

still_speed: float = 0.03

still_ang_speed class-attribute instance-attribute

still_ang_speed: float = 0.3

min_still_sec class-attribute instance-attribute

min_still_sec: float = Field(0.1667, ge=0)

disk_height class-attribute instance-attribute

disk_height: float = 0.01

phantom_in_disk_frac class-attribute instance-attribute

phantom_in_disk_frac: float = 0.9

phantom_z class-attribute instance-attribute

phantom_z: float = 0.1

consolidate_z class-attribute instance-attribute

consolidate_z: float = 0.05

rest_band class-attribute instance-attribute

rest_band: tuple[float, float] = (-0.02, 0.25)

contact_z class-attribute instance-attribute

contact_z: float = 0.02

FDROffDiskUntouchedOptions

FDROffDiskUntouchedOptions(**data)

Bases: ConfigurableOptions

Reject: an untouched object floats where nothing supports it for frames frames below max_height.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

frames class-attribute instance-attribute

frames: int = Field(5, ge=1)

max_height class-attribute instance-attribute

max_height: float = 0.15

FDRWristAccSpikeOptions

FDRWristAccSpikeOptions(**data)

Bases: ConfigurableOptions

Reject: the smooth-frame averaged MANO wrist accelerates faster than max_acc (m/s^2).

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

max_acc class-attribute instance-attribute

max_acc: float = 60.0

smooth class-attribute instance-attribute

smooth: int = Field(3, ge=1)

FDRObjBelowDeskOptions

FDRObjBelowDeskOptions(**data)

Bases: ConfigurableOptions

Reject: an untouched object starts or ends more than -min_z below the desk.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

min_z class-attribute instance-attribute

min_z: float = -0.05

FDRTooShortOptions

FDRTooShortOptions(**data)

Bases: ConfigurableOptions

Reject: the clip is shorter than min_sec seconds.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

min_sec class-attribute instance-attribute

min_sec: float = 2.5

FDRFirstUnsupportedOptions

FDRFirstUnsupportedOptions(**data)

Bases: ConfigurableOptions

Reject: an object is not over its own base (base_band of its lowest point, + slack) at the start.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

base_band class-attribute instance-attribute

base_band: float = 0.01

slack class-attribute instance-attribute

slack: float = 0.005

FDRFilterOptions

FDRFilterOptions(**data)

Bases: ConfigurableOptions

Reject rules over the imported clips (report.csv).

PARAMETER	DESCRIPTION
enable	False keeps every clip. TYPE: bool
contact	Distance (m) at which a fingertip or another object touches an object. TYPE: float
edge_frames	Frames at either end the start/end rules look at. TYPE: int
obj_offdisk_untouched	An untouched object floats where nothing supports it. TYPE: FDROffDiskUntouchedOptions
wrist_acc_spike	The MANO wrist accelerates too fast. TYPE: FDRWristAccSpikeOptions
obj_below_desk	An untouched object starts or ends below the desk. TYPE: FDRObjBelowDeskOptions
too_short	The clip is too short. TYPE: FDRTooShortOptions
first_unsupported	An object topples in the first frames. TYPE: FDRFirstUnsupportedOptions

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

enable class-attribute instance-attribute

enable: bool = True

contact class-attribute instance-attribute

contact: float = Field(0.015, gt=0)

edge_frames class-attribute instance-attribute

edge_frames: int = Field(5, ge=1)

obj_offdisk_untouched class-attribute instance-attribute

obj_offdisk_untouched: FDROffDiskUntouchedOptions = Field(default_factory=FDROffDiskUntouchedOptions)

wrist_acc_spike class-attribute instance-attribute

wrist_acc_spike: FDRWristAccSpikeOptions = Field(default_factory=FDRWristAccSpikeOptions)

obj_below_desk class-attribute instance-attribute

obj_below_desk: FDRObjBelowDeskOptions = Field(default_factory=FDRObjBelowDeskOptions)

too_short class-attribute instance-attribute

too_short: FDRTooShortOptions = Field(default_factory=FDRTooShortOptions)

first_unsupported class-attribute instance-attribute

first_unsupported: FDRFirstUnsupportedOptions = Field(default_factory=FDRFirstUnsupportedOptions)

FDRCoacdParams

FDRCoacdParams(**data)

Bases: ConfigurableOptions

Keyword arguments of coacd.run_coacd (see CoACD's documentation).

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

threshold class-attribute instance-attribute

threshold: float = Field(0.07, gt=0)

max_convex_hull class-attribute instance-attribute

max_convex_hull: int = Field(32, ge=-1)

preprocess_mode class-attribute instance-attribute

preprocess_mode: Literal['auto', 'on', 'off'] = 'auto'

preprocess_resolution class-attribute instance-attribute

preprocess_resolution: int = 50

resolution class-attribute instance-attribute

resolution: int = 2000

mcts_nodes class-attribute instance-attribute

mcts_nodes: int = 20

mcts_iterations class-attribute instance-attribute

mcts_iterations: int = 2000

mcts_max_depth class-attribute instance-attribute

mcts_max_depth: int = 5

pca class-attribute instance-attribute

pca: bool = False

merge class-attribute instance-attribute

merge: bool = True

decimate class-attribute instance-attribute

decimate: bool = False

max_ch_vertex class-attribute instance-attribute

max_ch_vertex: int = 256

extrude class-attribute instance-attribute

extrude: bool = False

apx_mode class-attribute instance-attribute

apx_mode: Literal['ch', 'box'] = 'ch'

FDRCoacdConvexOptions

FDRCoacdConvexOptions(**data)

Bases: ConfigurableOptions

CoACD convex decomposition on CPU workers.

PARAMETER	DESCRIPTION
force	Redo objects whose convex parts were made from their current mesh and these settings (parts made from any other are redone anyway). TYPE: bool
decimate_faces	Decimate a mesh with more faces first (dense scans make CoACD very slow); 0 never. TYPE: int
seed	CoACD's random seed. TYPE: int
coacd	coacd.run_coacd keyword arguments. TYPE: FDRCoacdParams

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

name class-attribute instance-attribute

name: Literal['coacd'] = 'coacd'

force class-attribute instance-attribute

force: bool = False

decimate_faces class-attribute instance-attribute

decimate_faces: int = Field(0, ge=0)

seed class-attribute instance-attribute

seed: int = 1

coacd class-attribute instance-attribute

coacd: FDRCoacdParams = Field(default_factory=FDRCoacdParams)

FDRCuacdConvexOptions

FDRCuacdConvexOptions(**data)

Bases: ConfigurableOptions

CuACD convex decomposition on GPUs (one process per GPU, about 20 GB each), then CoACD's merge.

PARAMETER	DESCRIPTION
force	Redo objects whose convex parts were made from their current mesh and these settings (parts made from any other are redone anyway). TYPE: bool
decimate_faces	Decimate a mesh with more faces first; 0 never. TYPE: int
gpus	Indices into the visible GPUs; None uses every one. TYPE: list[int] | None
threshold	Concavity threshold. TYPE: float
depth	Lookahead depth. TYPE: int
preprocess_resolution	Remeshing resolution for a mesh CuACD cannot take as is. TYPE: int
max_convex_hull	Parts kept after the merge. TYPE: int

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

name class-attribute instance-attribute

name: Literal['cuacd'] = 'cuacd'

force class-attribute instance-attribute

force: bool = False

decimate_faces class-attribute instance-attribute

decimate_faces: int = Field(0, ge=0)

gpus class-attribute instance-attribute

gpus: list[Annotated[int, Field(ge=0)]] | None = Field(None, min_length=1)

threshold class-attribute instance-attribute

threshold: float = Field(0.05, gt=0)

depth class-attribute instance-attribute

depth: int = Field(3, ge=1)

preprocess_resolution class-attribute instance-attribute

preprocess_resolution: int = 128

max_convex_hull class-attribute instance-attribute

max_convex_hull: int = Field(32, ge=1)

FDRApproachOptions

FDRApproachOptions(**data)

Bases: ConfigurableOptions

The first IK attempt starts from outside the object instead of from random restarts.

PARAMETER	DESCRIPTION
enable	Use the approach for attempt 0. TYPE: bool
backoff	Distance (m) behind the wrist target (the back of the hand) the hand starts at. TYPE: float
slide_frames	Frames over which the hand slides onto the wrist targets. TYPE: int
close_frames	Frames over which the fingertip targets move onto frame 0's. TYPE: int
preshape	Fraction of the contact-free start search's finger shape to begin with; 0 opens the hand. TYPE: float

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

enable class-attribute instance-attribute

enable: bool = True

backoff class-attribute instance-attribute

backoff: float = Field(0.06, ge=0)

slide_frames class-attribute instance-attribute

slide_frames: int = Field(30, ge=1)

close_frames class-attribute instance-attribute

close_frames: int = Field(30, ge=1)

preshape class-attribute instance-attribute

preshape: float = Field(1.0, ge=0, le=1)

FDREqualityOptions

FDREqualityOptions(**data)

Bases: ConfigurableOptions

A MuJoCo equality constraint pulling a robot site onto its mocap target.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

torque instance-attribute

torque: float

solref instance-attribute

solref: tuple[float, float]

solimp instance-attribute

solimp: tuple[float, float, float, float, float]

FDRConstraintsOptions

FDRConstraintsOptions(**data)

Bases: ConfigurableOptions

The IK's equality constraints: palm weld, fingertip connects and object weld.

PARAMETER	DESCRIPTION
palm	Weld of the palm site onto the MANO wrist. TYPE: FDREqualityOptions
tip	Connect of each fingertip site onto its MANO fingertip. TYPE: FDREqualityOptions
tip_width_ring_pinky	Factor on the tip solimp width for the ring and pinky (they track MANO less closely). TYPE: float
object	Stiff weld of each object onto its reference pose. TYPE: FDREqualityOptions

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

palm class-attribute instance-attribute

palm: FDREqualityOptions = Field(default_factory=lambda: FDREqualityOptions(torque=10.0, solref=(0.02, 1.0), solimp=(0.0, 0.95, 10.0, 0.5, 2.0)))

tip class-attribute instance-attribute

tip: FDREqualityOptions = Field(default_factory=lambda: FDREqualityOptions(torque=1.0, solref=(0.01, 1.0), solimp=(0.0, 0.95, 0.01, 0.5, 2.0)))

tip_width_ring_pinky class-attribute instance-attribute

tip_width_ring_pinky: float = 3.0

object class-attribute instance-attribute

object: FDREqualityOptions = Field(default_factory=lambda: FDREqualityOptions(torque=10.0, solref=(0.002, 1.0), solimp=(0.9, 0.95, 0.001, 0.5, 2.0)))

FDRContactOptions

FDRContactOptions(**data)

Bases: ConfigurableOptions

Contact settings of the IK model. mjENBL_OVERRIDE is on (as in FlashDexRetarget), so every contact, the explicit hand/object/support pairs included, takes o_solref, o_solimp and o_friction.

PARAMETER	DESCRIPTION
impratio	Ratio of frictional to normal constraint impedance. TYPE: float
o_solref	solref of every contact. TYPE: tuple[float, float]
o_solimp	solimp of every contact. TYPE: tuple[float, float, float, float, float]
o_friction	Friction of every contact (sliding, sliding, torsional, rolling, rolling); MuJoCo's default. TYPE: tuple[float, float, float, float, float]

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

impratio class-attribute instance-attribute

impratio: float = 1.0

o_solref class-attribute instance-attribute

o_solref: tuple[float, float] = (0.02, 1.0)

o_solimp class-attribute instance-attribute

o_solimp: tuple[float, float, float, float, float] = (0.95, 0.99, 0.002, 0.5, 2.0)

o_friction class-attribute instance-attribute

o_friction: tuple[float, float, float, float, float] = (1.0, 1.0, 0.005, 0.0001, 0.0001)

FDRIKSceneOptions

FDRIKSceneOptions(**data)

Bases: ConfigurableOptions

The IK model built around the robot MJCF.

PARAMETER	DESCRIPTION
armature	Armature of every robot joint (heavy joints keep the IK stable). TYPE: float
kp_wrist	Position gain of the forearm actuators (critically damped at qpos0). TYPE: float
kp_finger	Position gain of the finger actuators (critically damped at qpos0). TYPE: float
desk_depth	Minimum thickness of a support-disk cylinder. TYPE: float
object_density	Density of the object bodies. TYPE: float
object_armature	Free-joint armature of the objects. TYPE: float
object_frictionloss	Free-joint friction loss of the objects. TYPE: float
The

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

armature class-attribute instance-attribute

armature: float = 1.0

kp_wrist class-attribute instance-attribute

kp_wrist: float = 1000.0

kp_finger class-attribute instance-attribute

kp_finger: float = 300.0

desk_depth class-attribute instance-attribute

desk_depth: float = 0.3

object_density class-attribute instance-attribute

object_density: float = 1000.0

object_armature class-attribute instance-attribute

object_armature: float = 0.0001

object_frictionloss class-attribute instance-attribute

object_frictionloss: float = 0.0001

FDRMaxAccOptions

FDRMaxAccOptions(**data)

Bases: ConfigurableOptions

Flag: the palm accelerates faster than max_acc (m/s^2).

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

max_acc class-attribute instance-attribute

max_acc: float = 60.0

FDRMaxDepthOptions

FDRMaxDepthOptions(**data)

Bases: ConfigurableOptions

Flag: a depth beyond max_depth (m).

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

max_depth instance-attribute

max_depth: float

FDRTipTrackingOptions

FDRTipTrackingOptions(**data)

Bases: ConfigurableOptions

Flag: more than max_frac of the fingertip samples are max_dist (m) from MANO.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

max_dist class-attribute instance-attribute

max_dist: float = 0.03

max_frac class-attribute instance-attribute

max_frac: float = 0.35

FDRHandDeskPenetrationOptions

FDRHandDeskPenetrationOptions(**data)

Bases: ConfigurableOptions

Flag: the hand is max_depth (m) inside a disk for frames frames after frame after.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

max_depth class-attribute instance-attribute

max_depth: float = 0.02

frames class-attribute instance-attribute

frames: int = 5

after class-attribute instance-attribute

after: int = 30

FDRQualityOptions

FDRQualityOptions(**data)

Bases: ConfigurableOptions

Flags (report.csv) on a retargeted clip; a flagged clip is still packed.

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

palm_acc_spike class-attribute instance-attribute

palm_acc_spike: FDRMaxAccOptions = Field(default_factory=FDRMaxAccOptions)

hand_far_below_desk class-attribute instance-attribute

hand_far_below_desk: FDRMaxDepthOptions = Field(default_factory=lambda: FDRMaxDepthOptions(max_depth=0.08))

tip_tracking class-attribute instance-attribute

tip_tracking: FDRTipTrackingOptions = Field(default_factory=FDRTipTrackingOptions)

hand_obj_penetration class-attribute instance-attribute

hand_obj_penetration: FDRMaxDepthOptions = Field(default_factory=lambda: FDRMaxDepthOptions(max_depth=0.02))

hand_desk_penetration class-attribute instance-attribute

hand_desk_penetration: FDRHandDeskPenetrationOptions = Field(default_factory=FDRHandDeskPenetrationOptions)

FDRRetargetOptions

FDRRetargetOptions(**data)

Bases: ConfigurableOptions

MuJoCo physics IK: the robot's palm and fingertip sites follow the MANO wrist and fingertips while the hands collide with the objects and support disks.

PARAMETER	DESCRIPTION
force	Redo clips whose <robot>/retargeted/<clip>.npz was made from this run's inputs and settings (a clip made from any other is redone anyway). TYPE: bool
sim_dt	MuJoCo timestep. TYPE: float
solver_iterations	MuJoCo solver iterations. TYPE: int
solver_ls_iterations	MuJoCo line-search iterations. TYPE: int
n_guess	Random restarts of a start search. TYPE: int
guess_steps	Steps each restart runs. TYPE: int
guess_jitter	Noise (m) on the fingertip targets of a restart. TYPE: float
start_settle	Frame-0 steps before the clip runs. TYPE: int
approach	Attempt 0's start from outside the object. TYPE: FDRApproachOptions
settle_steps	Steps per frame with the fingertip constraints off and the joints PD-held (contacts push out). TYPE: int
recover_tip_err	A restart is searched when the recent mean fingertip error exceeds this (m). TYPE: float
recover_window	Frames that error averages over. TYPE: int
recover_cooldown	Frames between restarts. TYPE: int
attempts	IK runs per clip; the best is kept. TYPE: int
pen_weight	Weight of a run's max hand-object penetration against its mean fingertip error. TYPE: float
smooth	Moving-average window, centred on each frame, over the kept run (the first smooth frames are cropped). TYPE: int
constraints	Palm, fingertip and object equality constraints. TYPE: FDRConstraintsOptions
contact	Global contact settings. TYPE: FDRContactOptions
scene	The IK model around the robot MJCF. TYPE: FDRIKSceneOptions
quality	Quality flags of the kept run. TYPE: FDRQualityOptions

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

force class-attribute instance-attribute

force: bool = False

sim_dt class-attribute instance-attribute

sim_dt: float = Field(0.002083, gt=0)

solver_iterations class-attribute instance-attribute

solver_iterations: int = Field(20, ge=1)

solver_ls_iterations class-attribute instance-attribute

solver_ls_iterations: int = Field(50, ge=1)

n_guess class-attribute instance-attribute

n_guess: int = Field(48, ge=1)

guess_steps class-attribute instance-attribute

guess_steps: int = Field(30, ge=1)

guess_jitter class-attribute instance-attribute

guess_jitter: float = Field(0.002, ge=0)

start_settle class-attribute instance-attribute

start_settle: int = Field(60, ge=0)

approach class-attribute instance-attribute

approach: FDRApproachOptions = Field(default_factory=FDRApproachOptions)

settle_steps class-attribute instance-attribute

settle_steps: int = Field(80, ge=0)

recover_tip_err class-attribute instance-attribute

recover_tip_err: float = Field(0.045, gt=0)

recover_window class-attribute instance-attribute

recover_window: int = Field(15, ge=1)

recover_cooldown class-attribute instance-attribute

recover_cooldown: int = Field(30, ge=0)

attempts class-attribute instance-attribute

attempts: int = Field(6, ge=1)

pen_weight class-attribute instance-attribute

pen_weight: float = Field(0.5, ge=0)

smooth class-attribute instance-attribute

smooth: int = Field(3, ge=1)

constraints class-attribute instance-attribute

constraints: FDRConstraintsOptions = Field(default_factory=FDRConstraintsOptions)

contact class-attribute instance-attribute

contact: FDRContactOptions = Field(default_factory=FDRContactOptions)

scene class-attribute instance-attribute

scene: FDRIKSceneOptions = Field(default_factory=FDRIKSceneOptions)

quality class-attribute instance-attribute

quality: FDRQualityOptions = Field(default_factory=FDRQualityOptions)

FDRPackOptions

FDRPackOptions(**data)

Bases: ConfigurableOptions

motion.pt packing.

PARAMETER	DESCRIPTION
contact_threshold	A fingertip within this distance (m) of the object mesh is in contact (dilated by one frame). TYPE: float
vel_sigma	Gaussian smoothing (frames) of the finite-difference velocities. TYPE: float

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

contact_threshold class-attribute instance-attribute

contact_threshold: float = Field(0.015, gt=0)

vel_sigma class-attribute instance-attribute

vel_sigma: float = Field(2.0, ge=0)

FDRSdfOptions

FDRSdfOptions(**data)

Bases: ConfigurableOptions

Object SDF grids over [-extent, extent]^3 in the object frame (signed distance + gradient).

PARAMETER	DESCRIPTION
grid_n	Grid points per axis; must match the training task's SDF settings. TYPE: int
extent	Half-size (m) of the grid; must match the training task's SDF settings. TYPE: float
device	Where warp bakes. TYPE: Literal['cuda', 'cpu']
gpus	Indices into the visible GPUs; None uses every one. TYPE: list[int] | None
workers_per_gpu	Processes per GPU. TYPE: int

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

grid_n class-attribute instance-attribute

grid_n: int = Field(128, ge=2)

extent class-attribute instance-attribute

extent: float = Field(0.3, gt=0)

device class-attribute instance-attribute

device: Literal['cuda', 'cpu'] = 'cuda'

gpus class-attribute instance-attribute

gpus: list[Annotated[int, Field(ge=0)]] | None = Field(default_factory=lambda: [0], min_length=1)

workers_per_gpu class-attribute instance-attribute

workers_per_gpu: int = Field(8, ge=1)

FDRKinematicRetargetingOptions

FDRKinematicRetargetingOptions(**data)

Bases: ConfigurableOptions

Imported clips -> <dataset>/<robot>/motion.pt for one robot.

PARAMETER	DESCRIPTION
dataset	The dataset directory (human_demo/ and objects/ from the import). TYPE: str
robot	The bimanual robot the clips are retargeted to. TYPE: Literal['xhand', 'sharpa']
workers	Processes; None uses half the CPU cores. TYPE: int | None
steps	Which steps run. TYPE: FDRStepsOptions
desk	Support-disk fitting. TYPE: FDRDeskOptions
filters	Reject rules. TYPE: FDRFilterOptions
convex	Convex decomposition backend (name: coacd or cuacd). TYPE: FDRConvexOptions
retarget	Physics IK. TYPE: FDRRetargetOptions
pack	motion.pt packing. TYPE: FDRPackOptions
sdf	Object SDF grids. TYPE: FDRSdfOptions

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

dataset instance-attribute

dataset: str

robot class-attribute instance-attribute

robot: Literal['xhand', 'sharpa'] = 'xhand'

workers class-attribute instance-attribute

workers: int | None = Field(None, ge=1)

steps class-attribute instance-attribute

steps: FDRStepsOptions = Field(default_factory=FDRStepsOptions)

desk class-attribute instance-attribute

desk: FDRDeskOptions = Field(default_factory=FDRDeskOptions)

filters class-attribute instance-attribute

filters: FDRFilterOptions = Field(default_factory=FDRFilterOptions)

convex class-attribute instance-attribute

convex: FDRConvexOptions = Field(default_factory=FDRCoacdConvexOptions)

retarget class-attribute instance-attribute

retarget: FDRRetargetOptions = Field(default_factory=FDRRetargetOptions)

pack class-attribute instance-attribute

pack: FDRPackOptions = Field(default_factory=FDRPackOptions)

sdf class-attribute instance-attribute

sdf: FDRSdfOptions = Field(default_factory=FDRSdfOptions)

Import: a released dataset -> <dataset>/human_demo/<clip>.npz + <dataset>/objects/<id>/visual.obj.

The release is read by its :mod:eden.datasets loader (TacoDataset, OakInk2Dataset, Hot3dDataset, DexYCBDataset) with its MANO parameters (body_model_params), in the loader's z-up world at the release's frame rate; the loader cuts the clips and picks the object each hand holds. Per clip, this runs MANO, moves the surface the objects rest on to table_z, takes the wrist and fingertip reference the rewards compare against, and resamples to fps. A clip is skipped when an object flips more than max_obj_rot_step_deg in one source frame.

Attributes

SOURCE_DATASETS module-attribute

SOURCE_DATASETS = {'taco': 'taco_dataset', 'oakink2': 'oak_ink2_dataset', 'hot3d': 'hot3d_dataset', 'dexycb': 'dex_ycb_dataset'}

MESH_SOURCE_FILE module-attribute

MESH_SOURCE_FILE = 'visual.json'

Classes

Functions:

open_dataset

open_dataset(cfg: FDRImportOptions) -> DatasetBase

The :mod:eden.datasets loader of cfg.source, built over cfg.raw with its MANO parameters.

Source code in eden/extensions/retargeting/flashdexretarget/importer.py

def open_dataset(cfg: FDRImportOptions) -> DatasetBase:
    """The :mod:`eden.datasets` loader of ``cfg.source``, built over ``cfg.raw`` with its MANO parameters."""
    from eden.datasets import DATASET_REGISTRY

    cls = DATASET_REGISTRY.get(SOURCE_DATASETS[cfg.source.name])
    params: dict[str, Any] = {
        "raw_data_path": cfg.raw,
        "body_model_params": True,
        "body_model_dir": cfg.mano_dir,
    }
    if cfg.source.name == "dexycb":
        from eden.datasets.dex_ycb_dataset import DEX_YCB_FRAMERATE

        raw = Path(cfg.raw)
        params |= {
            "target_framerate": DEX_YCB_FRAMERATE,
            "calibration_dir": str(raw / "calibration"),
            "models_dir": str(raw / "models"),
            # FlashDexRetarget fits its own support disks on the release's poses and meshes.
            "ground_initial_pose": False,
            "apply_mesh_correction": False,
        }
    else:
        params |= {"target_framerate": cls.source_framerate, "sides": cfg.sides}
    taken = sorted(set(cfg.source.dataset) & set(params))
    if taken:
        raise ValueError(
            f"source.dataset sets {taken}, which the import sets itself (from raw, mano_dir, sides and the release's "
            f"frame rate). Set them on the import options instead, or drop them from source.dataset."
        )
    return cls(cls.configure(**params, **cfg.source.dataset), torch.device("cpu"))

import_demos

import_demos(cfg: FDRImportOptions) -> list[dict]

cfg.raw (a released dataset, cfg.source) -> <dataset>/human_demo/<clip>.npz and <dataset>/objects/<id>/visual.obj.

PARAMETER	DESCRIPTION
cfg	Import settings. TYPE: FDRImportOptions

RETURNS	DESCRIPTION
list[dict]	One row per clip: its name and frame count and objects, or the reason it was skipped (skip).

Source code in eden/extensions/retargeting/flashdexretarget/importer.py

def import_demos(cfg: FDRImportOptions) -> list[dict]:
    """``cfg.raw`` (a released dataset, ``cfg.source``) -> ``<dataset>/human_demo/<clip>.npz`` and
    ``<dataset>/objects/<id>/visual.obj``.

    Parameters
    ----------
    cfg : FDRImportOptions
        Import settings.

    Returns
    -------
    list[dict]
        One row per clip: its name and frame count and objects, or the reason it was skipped (``skip``).
    """
    mano_dir = Path(cfg.mano_dir)
    for f in ("MANO_RIGHT.pkl", "MANO_LEFT.pkl"):
        if not (mano_dir / f).exists():
            raise FileNotFoundError(
                f"{mano_dir / f} missing. The import needs the MANO hand models, which are not redistributable: "
                "register at mano.is.tue.mpg.de, download 'Models & Code' (mano_v1_2.zip) and point `mano_dir` at "
                "its models/ directory."
            )
    en.logger.info(f"[import] {cfg.source.name}: reading {cfg.raw}")
    dataset = open_dataset(cfg)
    clips, skipped = _clips(dataset, cfg)
    out_dir = Path(cfg.dataset) / "human_demo"
    out_dir.mkdir(parents=True, exist_ok=True)
    before = {p.stem for p in C.list_clips(out_dir)}
    rows = [{"clip": name, "skip": reason} for name, reason in skipped]
    done = 0
    for recording, group in itertools.groupby(sorted(clips, key=lambda c: c.recording), key=lambda c: c.recording):
        group = list(group)
        rec = _recording(dataset, recording, cfg, group[0].fps)
        rows += [_write(dataset, clip, cfg, out_dir, rec) for clip in group]
        if (done + len(group)) // 200 > done // 200 or done + len(group) == len(clips):
            en.logger.info(f"[import] {done + len(group)}/{len(clips)} clips")
        done += len(group)
    kept = [r for r in rows if "skip" not in r]
    reasons = count_reasons(r["skip"] for r in rows if "skip" in r)
    if not kept:
        # Before any earlier clip is removed below: an import that keeps nothing is a misconfiguration, not a release
        # whose clips all went away.
        raise RuntimeError(
            f"[import] every clip of {cfg.raw} was skipped ({len(rows)}: {reasons}), so nothing was written to "
            f"{out_dir} and its earlier clips are left as they are. First skips: {[r['skip'] for r in rows[:3]]}. "
            f"Check `sides` (now {cfg.sides!r}: each of those hands must hold an object) and the source's settings."
        )
    exported: dict[str, list[str]] = {}
    for oid in sorted({o for r in kept for k in ("objects", "context") if r.get(k) for o in r[k].split()}):
        status = _export_mesh(dataset, oid, Path(cfg.dataset) / "objects" / oid, cfg.textures)
        exported.setdefault(status, []).append(oid)
        if cfg.textures:
            # A trimesh mesh sits in reference cycles, so the textured one just exported is freed by the cycle
            # collector, which runs on object counts, not bytes: without this, the decoded textures (~50 MB each at
            # 4096^2) pile up, 1.5 GB over DexYCB's 20 objects.
            gc.collect()
    objects = Path(cfg.dataset) / "objects"
    changed, rewritten = exported.get("changed", []), exported.get("rewritten", [])
    retextured = exported.get("retextured", [])
    if retextured:
        en.logger.info(
            f"[import] {len(retextured)} objects already in {objects} got this release's materials and textures, their "
            f"visual.obj (and the convex parts and SDF grids made from it) unchanged, e.g. {retextured[:3]}"
        )
    if rewritten:
        en.logger.warning(
            f"[import] {len(rewritten)} objects already in {objects} have this release's geometry, but their "
            f"visual.obj was written again (other UVs, or another trimesh version's formatting), e.g. "
            f"{rewritten[:3]}; their convex parts, SDF grids and retargeted clips are redone."
        )
    if changed:
        en.logger.warning(
            f"[import] {len(changed)} objects already in {objects} had another mesh, now replaced "
            f"by this release's, e.g. {changed[:3]}; their convex parts, SDF grids and retargeted clips are redone."
        )
    en.logger.info(f"[import] {len(kept)} clips -> {out_dir}, skipped {len(rows) - len(kept)} {reasons}")
    # A clip this import skips but an earlier one wrote would still be retargeted: it goes.
    stale = sorted(before & {r["clip"] for r in rows if "skip" in r})
    for name in stale:
        (out_dir / f"{name}.npz").unlink()
    if stale:
        en.logger.info(
            f"[import] removed {len(stale)} clips of an earlier import that this one skips, e.g. {stale[:3]}"
        )
    others = sorted(before - {r["clip"] for r in rows})
    if others:
        en.logger.warning(
            f"[import] {out_dir} also holds {len(others)} clips this import did not write (another release or other "
            f"source settings), e.g. {others[:3]}; `eden retarget flashdex kinematic` retargets them too. Delete them "
            "if they are left over."
        )
    return rows

Kinematic retargeting: imported clips -> <dataset>/<robot>/motion.pt for one robot.

================= ================================================== =========================================== step does writes (under the dataset) ================= ================================================== =========================================== preprocess support disks, reject rules <robot>/report.csv convex convex decomposition (CoACD or CuACD) objects/<id>/convex/*.obj retarget MuJoCo physics IK <robot>/retargeted/<clip>.npz pack kept clips into one file <robot>/motion.pt urdf object URDFs for simulation objects/<id>/object.urdf sdf object SDF grids objects/<id>/.sdf_cache/ ================= ================================================== ===========================================

Classes

Functions:

run_kinematic_retargeting

run_kinematic_retargeting(cfg: FDRKinematicRetargetingOptions) -> Path | None

Run the enabled steps over cfg.dataset.

The robot's outputs go to <dataset>/<robot>/ (report.csv, retargeted/, motion.pt), so each robot retargeted from one import keeps its own; human_demo/ and objects/ are shared. A dataset without human_demo/ but with the robot's motion.pt only gets its objects prepared (convex, sdf).

PARAMETER	DESCRIPTION
cfg	Pipeline settings. TYPE: FDRKinematicRetargetingOptions

RETURNS	DESCRIPTION
Path | None	<dataset>/<robot>/motion.pt when it exists at the end, else None.

Source code in eden/extensions/retargeting/flashdexretarget/pipeline.py

def run_kinematic_retargeting(cfg: FDRKinematicRetargetingOptions) -> Path | None:
    """Run the enabled steps over ``cfg.dataset``.

    The robot's outputs go to ``<dataset>/<robot>/`` (``report.csv``, ``retargeted/``, ``motion.pt``), so each robot
    retargeted from one import keeps its own; ``human_demo/`` and ``objects/`` are shared. A dataset without
    ``human_demo/`` but with the robot's ``motion.pt`` only gets its objects prepared (convex, sdf).

    Parameters
    ----------
    cfg : FDRKinematicRetargetingOptions
        Pipeline settings.

    Returns
    -------
    Path | None
        ``<dataset>/<robot>/motion.pt`` when it exists at the end, else ``None``.
    """
    workers = cfg.workers or max(1, (os.cpu_count() or 2) // 2)  # half the CPU cores by default
    steps = cfg.steps
    # What the last steps need, checked before the hours of IK ahead of them.
    if steps.convex and cfg.convex.name == "cuacd":
        check_cuacd(cfg.convex)
    if steps.sdf:
        check_sdf(cfg.sdf)
    dataset = Path(cfg.dataset)
    # Per robot: retargeting another robot keeps this one's results, and two robots' runs never share a file.
    demo, out = dataset / "human_demo", dataset / cfg.robot
    retargeted, motion, report = out / "retargeted", out / "motion.pt", out / "report.csv"
    old = [p for p in (dataset / "motion.pt", dataset / "report.csv", dataset / "retargeted") if p.exists()]
    if old:  # the layout before the per-robot directories: a training run pointed at them would load the stale ones
        en.logger.warning(
            f"[kinematic_retargeting] {dataset} still holds {[p.name for p in old]} of an earlier version, which wrote "
            f"one robot's outputs into the dataset itself; this run writes {out}/. Delete them."
        )
    from_clips = demo.is_dir()
    if from_clips:
        fps = clip_fps(list_clips(demo))
        en.logger.info(f"[kinematic_retargeting] preprocess: {demo} ({fps:g} Hz)")
        out.mkdir(exist_ok=True)
        rows, kept = preprocess_clips(demo, dataset, cfg, fps, workers)
        _write_report(report, rows)
        if not kept:
            raise RuntimeError(f"Every clip was rejected; the reasons are in {report} (column 'reject').")
        ids = object_ids(kept)
    elif motion.exists():
        en.logger.info(f"[kinematic_retargeting] no {demo}: preparing the objects of {motion}")
        packed = torch.load(motion, map_location="cpu", weights_only=True)
        ids = sorted({mesh.parent.name for mesh, _ in object_meshes(packed, motion.parent)})
    else:
        raise FileNotFoundError(
            f"{dataset} has neither human_demo/ nor {cfg.robot}/motion.pt; import a release into it first with "
            "`eden retarget flashdex import`."
        )

    if steps.convex:
        en.logger.info(f"[kinematic_retargeting] convex: {len(ids)} objects (<object>/convex/, {cfg.convex.name})")
        decompose_objects(dataset, ids, cfg.convex, workers)
    if from_clips and (steps.retarget or steps.pack):
        keys = _cache_keys(kept, dataset, cfg)
        names = set(keys)
        if steps.retarget:
            en.logger.info(f"[kinematic_retargeting] retarget: {len(kept)} clips -> {retargeted} ({cfg.robot})")
            ik = {
                r["clip"]: r
                for r in retarget_clips(
                    list(kept.items()), dataset, retargeted, cfg.robot, cfg.retarget, fps, workers, keys
                )
            }
            for r in rows:
                r.update(ik.get(r["clip"], {}))
            _write_report(report, rows)
            failed = {name for name, r in ik.items() if r.get("error")}
            if failed == names:
                raise RuntimeError(f"Every clip failed to retarget; the errors are in {report} (column 'error').")
            names -= failed
        if steps.pack:
            en.logger.info(f"[kinematic_retargeting] pack: {retargeted} -> {motion}")
            build(retargeted, dataset, motion, fps, cfg.pack, cfg.robot, {n: keys[n] for n in sorted(names)})
    if steps.urdf and motion.exists():
        packed = torch.load(motion, map_location="cpu", weights_only=True)
        write_object_urdfs(dataset, [(mesh.parent.name, scale) for mesh, scale in object_meshes(packed, motion.parent)])
    if steps.sdf and not motion.exists():
        en.logger.info(f"[kinematic_retargeting] sdf: skipped, no {motion} yet")
    elif steps.sdf:
        en.logger.info(f"[kinematic_retargeting] sdf: <object>/.sdf_cache/, {cfg.sdf.grid_n}^3")
        bake(motion, cfg.sdf, workers)
    return motion if motion.exists() else None

Preprocess: support disks and reject rules over every imported clip (writes only the report rows).

Classes

Functions:

preprocess_clips

preprocess_clips(clips: Path, pool: Path, cfg: FDRKinematicRetargetingOptions, fps: float, workers: int) -> tuple[list[dict], dict[Path, ndarray | None]]

Report rows of every clip, and the kept clips with their support disks.

PARAMETER	DESCRIPTION
clips	The human_demo/ directory. TYPE: Path
pool	The dataset directory holding objects/. TYPE: Path
cfg	Pipeline settings (desk, filters). TYPE: FDRKinematicRetargetingOptions
fps	The clips' frame rate. TYPE: float
workers	Processes. TYPE: int

RETURNS	DESCRIPTION
tuple[list[dict], dict[Path, ndarray | None]]	One report row per clip (reject empty when kept), and kept clip path -> its (K, 5) support disks.

Source code in eden/extensions/retargeting/flashdexretarget/preprocess.py

def preprocess_clips(
    clips: Path, pool: Path, cfg: FDRKinematicRetargetingOptions, fps: float, workers: int
) -> tuple[list[dict], dict[Path, np.ndarray | None]]:
    """Report rows of every clip, and the kept clips with their support disks.

    Parameters
    ----------
    clips : Path
        The ``human_demo/`` directory.
    pool : Path
        The dataset directory holding ``objects/``.
    cfg : FDRKinematicRetargetingOptions
        Pipeline settings (``desk``, ``filters``).
    fps : float
        The clips' frame rate.
    workers : int
        Processes.

    Returns
    -------
    tuple[list[dict], dict[Path, np.ndarray | None]]
        One report row per clip (``reject`` empty when kept), and kept clip path -> its ``(K, 5)`` support disks.
    """
    paths = C.list_clips(clips)
    en.logger.info(f"[preprocess] {len(paths)} clips, desk={cfg.desk.mode}, filter={cfg.filters.enable}")
    jobs = [(str(p), str(pool), cfg, fps) for p in paths]
    with C.worker_pool(max(1, min(workers, len(jobs)))) as ex:
        results = list(ex.map(_process, *zip(*jobs))) if jobs else []

    rows = [r for r, _ in results]
    kept = {p: disks for p, (r, disks) in zip(paths, results) if not r["reject"]}
    counts: dict[str, int] = {}
    for r in rows:
        for reason in filter(None, r["reject"].split(";")):
            counts[reason] = counts.get(reason, 0) + 1
    en.logger.info(f"[preprocess] kept {len(kept)} / {len(rows)}  {counts}")
    return rows, kept

Retarget step: every IK attempt of every kept clip as a job of its own, the best attempt per clip saved to <robot>/retargeted/<clip>.npz and scored for the report.

A saved clip is a cache keyed (cache_key) on everything its IK read: the robot and its model files, the imported clip's bytes, its support disks, its objects' convex parts and the retarget options (force and the quality flags, which are scored afresh every run, aside), so a re-import, an updated asset, redone convex parts or a changed setting redoes it.

Functions:

cache_key

cache_key(clip_path: Path, disks: ndarray | None, robot: str, robot_sha1: str, convex_sha1: dict[str, str], cfg) -> dict

What a retargeted clip was made from (cfg: retarget); compared field by field against a saved clip's.

robot_sha1 is :func:~eden.extensions.retargeting.flashdexretarget.scene.robot_sha1 of the clip's hands and convex_sha1 maps each of its objects to :func:~eden.extensions.retargeting.flashdexretarget.convex.parts_sha1.

Source code in eden/extensions/retargeting/flashdexretarget/retarget.py

def cache_key(
    clip_path: Path, disks: np.ndarray | None, robot: str, robot_sha1: str, convex_sha1: dict[str, str], cfg
) -> dict:
    """What a retargeted clip was made from (cfg: retarget); compared field by field against a saved clip's.

    ``robot_sha1`` is :func:`~eden.extensions.retargeting.flashdexretarget.scene.robot_sha1` of the clip's hands and
    ``convex_sha1`` maps each of its objects to
    :func:`~eden.extensions.retargeting.flashdexretarget.convex.parts_sha1`.
    """
    return {
        "robot": robot,
        "robot_sha1": robot_sha1,
        "clip_sha1": hashlib.sha1(Path(clip_path).read_bytes()).hexdigest(),
        "support_disks_sha1": None
        if disks is None
        else hashlib.sha1(np.ascontiguousarray(disks, np.float64)).hexdigest(),
        "convex_sha1": dict(sorted(convex_sha1.items())),
        "retarget": cfg.model_dump(mode="json", exclude={"force", "quality"}),
    }

saved_key

saved_key(path: Path) -> dict | None

The cache_key a retargeted clip was saved with; None when it has none.

Source code in eden/extensions/retargeting/flashdexretarget/retarget.py

def saved_key(path: Path) -> dict | None:
    """The ``cache_key`` a retargeted clip was saved with; None when it has none."""
    with np.load(path, allow_pickle=False) as z:
        return json.loads(str(z["cache_key"])) if "cache_key" in z.files else None

key_mismatch

key_mismatch(saved: dict | None, key: dict) -> list[str]

The fields of key a saved clip's key differs in (['cache_key'] when it has none).

Source code in eden/extensions/retargeting/flashdexretarget/retarget.py

def key_mismatch(saved: dict | None, key: dict) -> list[str]:
    """The fields of ``key`` a saved clip's key differs in (``['cache_key']`` when it has none)."""
    if saved is None:
        return ["cache_key"]
    return [k for k in key if saved.get(k) != key[k]]

retarget_clips

retarget_clips(clips: list[tuple[Path, ndarray | None]], pool: Path, out: Path, robot: str, cfg, fps: float, workers: int, keys: dict[str, dict]) -> list[dict]

IK of (clip file, support disks) pairs into out, a cache: a clip whose saved result has its keys[stem] (:func:cache_key) is not redone unless cfg.force (cfg: retarget). Every IK run of every clip (cfg.attempts per clip) is a job of its own, so a long clip takes one run's time. Returns the report rows; retarget_seconds sums a clip's runs. A clip that fails has an error and no file in out.

Source code in eden/extensions/retargeting/flashdexretarget/retarget.py

def retarget_clips(
    clips: list[tuple[Path, np.ndarray | None]],
    pool: Path,
    out: Path,
    robot: str,
    cfg,
    fps: float,
    workers: int,
    keys: dict[str, dict],
) -> list[dict]:
    """IK of (clip file, support disks) pairs into out, a cache: a clip whose saved result has its ``keys[stem]``
    (:func:`cache_key`) is not redone unless cfg.force (cfg: retarget). Every IK run of every clip (cfg.attempts per
    clip) is a job of its own, so a long clip takes one run's time. Returns the report rows; retarget_seconds sums a
    clip's runs. A clip that fails has an ``error`` and no file in out.
    """
    out.mkdir(parents=True, exist_ok=True)
    en.logger.info(f"[retarget] {len(clips)} clips to {robot}")
    args = (str(pool), robot, cfg, fps)
    rows, runs, spent, jobs = [], {}, {}, {}

    def finish(p: Path, d, finished, seconds: float):
        return ex.submit(_finish, str(p), d, str(pool), str(out), robot, cfg, fps, finished, seconds, keys[p.stem])

    with C.worker_pool(max(1, workers)) as ex:
        for p, d in clips:
            try:
                check_length(C.num_frames(C.load_clip(p)), cfg.smooth)
            except ValueError as e:
                rows.append(_failed(out / p.name, {"clip": p.stem, "error": str(e)}, len(rows) + 1, len(clips)))
            else:
                if _needs_ik(keys[p.stem], out / p.name, cfg):
                    runs[p], spent[p] = [None] * cfg.attempts, 0.0
                    for k in range(cfg.attempts):
                        jobs[ex.submit(_attempt, str(p), d, *args, k)] = (p, d, k)
                else:
                    jobs[finish(p, d, None, 0.0)] = (p, d, None)
        while jobs:
            done, _ = wait(jobs, return_when=FIRST_COMPLETED)
            for fut in done:
                p, d, k = jobs.pop(fut)
                if k is None:
                    r = fut.result()
                    if r.get("error"):
                        rows.append(_failed(out / p.name, r, len(rows) + 1, len(clips)))
                        continue
                    rows.append(r)
                    msg = f"{r['source']}, tip {r.get('ik_tip_err_mean_m', float('nan')) * 1e3:.1f} mm {r.get('flags') or 'ok'}"
                    en.logger.info(
                        f"[retarget] {len(rows)}/{len(clips)} {r['clip']} ({r['retarget_seconds']} s): {msg}"
                    )
                    continue
                runs[p][k], seconds = fut.result()
                spent[p] += seconds
                if any(r is None for r in runs[p]):
                    continue
                finished = runs.pop(p)
                failures = [(i, r) for i, r in enumerate(finished) if isinstance(r, _Failure)]
                if failures:
                    distinct = list(dict.fromkeys(r.error for _, r in failures))
                    row = {
                        "clip": p.stem,
                        "error": f"{len(failures)}/{len(finished)} IK attempts raised: " + "; ".join(distinct),
                        "traceback": f"attempt {failures[0][0]}:\n{failures[0][1].traceback}",
                        "retarget_seconds": round(spent[p], 1),
                    }
                    rows.append(_failed(out / p.name, row, len(rows) + 1, len(clips)))
                    continue
                jobs[finish(p, d, finished, spent[p])] = (p, d, None)
    errors = sum(1 for r in rows if r.get("error"))
    flagged = sum(1 for r in rows if r.get("flags"))
    en.logger.info(f"[retarget] {len(rows) - errors} clips in {out} ({errors} errors, {flagged} flagged)")
    return rows

Pack: kept retargeted clips -> <dataset>/<robot>/motion.pt, with velocities and fingertip contact labels.

Per-frame arrays are concatenated over clips; clip i covers frames length_starts[i] to length_starts[i] + motion_num_frames[i]. joint_pos is in the robot MJCF's joint order, which joint_names lists; robot names the robot and fps the frame rate.

Functions:

velocity

velocity(x: ndarray, dt: float, sigma: float) -> ndarray

Finite difference, Gaussian-smoothed over time (sigma in frames).

Source code in eden/extensions/retargeting/flashdexretarget/pack.py

def velocity(x: np.ndarray, dt: float, sigma: float) -> np.ndarray:
    """Finite difference, Gaussian-smoothed over time (sigma in frames)."""
    v = np.gradient(x, axis=0) / dt
    return gaussian_filter1d(v, sigma=sigma, axis=0, mode="nearest").astype(np.float32)

angular_velocity

angular_velocity(rot: ndarray, dt: float, sigma: float) -> ndarray

World-frame angular velocity of a (T, 3, 3) rotation sequence.

Source code in eden/extensions/retargeting/flashdexretarget/pack.py

def angular_velocity(rot: np.ndarray, dt: float, sigma: float) -> np.ndarray:
    """World-frame angular velocity of a (T, 3, 3) rotation sequence."""
    w = Rotation.from_matrix(rot[1:] @ np.swapaxes(rot[:-1], -2, -1)).as_rotvec() / dt
    w = np.concatenate([w, w[-1:]], axis=0)
    return gaussian_filter1d(w, sigma=sigma, axis=0, mode="nearest").astype(np.float32)

tip_contacts

tip_contacts(tips: ndarray, pose: ndarray, verts: ndarray, tree: cKDTree, threshold: float) -> tuple[ndarray, ndarray, ndarray]

Fingertips (T, 5, 3) vs the object: distance to the nearest of its surface points verts (tree over them; :class:~eden.extensions.retargeting.flashdexretarget.clips.ObjectMeshes surface) (T, 5), contact flags (distance < threshold, dilated by one frame), and that point in the object frame (T, 5, 3).

Source code in eden/extensions/retargeting/flashdexretarget/pack.py

def tip_contacts(
    tips: np.ndarray, pose: np.ndarray, verts: np.ndarray, tree: cKDTree, threshold: float
) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    """Fingertips (T, 5, 3) vs the object: distance to the nearest of its surface points ``verts`` (``tree`` over
    them; :class:`~eden.extensions.retargeting.flashdexretarget.clips.ObjectMeshes` ``surface``) (T, 5), contact
    flags (distance < threshold, dilated by one frame), and that point in the object frame (T, 5, 3).
    """
    rot, pos = pose[:, :3, :3], pose[:, :3, 3]
    local = np.einsum("tji,tkj->tki", rot, tips - pos[:, None])  # R^T (tip - p)
    dist, idx = tree.query(local.reshape(-1, 3))
    dist = dist.reshape(local.shape[:2]).astype(np.float32)
    near = dist < threshold
    flags = near.copy()
    flags[1:] |= near[:-1]
    flags[:-1] |= near[1:]
    return dist, flags.astype(np.float32), verts[idx].reshape(local.shape).astype(np.float32)

load_clip

load_clip(path: Path, meshes: ObjectMeshes, fps: float, cfg) -> tuple[dict, dict]

Source code in eden/extensions/retargeting/flashdexretarget/pack.py

def load_clip(path: Path, meshes: C.ObjectMeshes, fps: float, cfg) -> tuple[dict, dict]:
    clip = C.load_clip(path)
    dt, sigma = 1.0 / fps, cfg.vel_sigma
    qpos = clip["qpos"].astype(np.float32)
    T = len(qpos)
    if T < 2:
        raise ValueError(f"{path.name}: needs at least 2 frames")
    sides = C.sides(clip)
    if not sides:
        raise ValueError(f"{path.name}: no mano_right_* / mano_left_* arrays")
    d = {"joint_pos": qpos, "joint_vel": velocity(qpos, dt, sigma)}
    meta: dict = {"sides": sides, "support": clip.get("support_disks")}
    tip_cols = [C.MANO_JOINT_NAMES.index(f"{f}_tip") for f in C.FINGERS]
    for s in sides:
        joints = C.joints(clip, s).astype(np.float32)
        wrist_pos = clip[f"mano_{s}_wrist_pos"].astype(np.float32)
        wrist_rot = clip[f"mano_{s}_wrist_rot"].astype(np.float32)
        pose = C.obj_pose(clip, s)
        for key, arr in (
            ("mano_joints", joints),
            ("wrist_pos", wrist_pos),
            ("wrist_rot", wrist_rot),
            ("obj_pose", pose),
        ):
            if len(arr) != T:
                raise ValueError(f"{path.name}: {s} {key} has {len(arr)} frames, qpos {T}")
        obj_id, scale = C.obj_id(clip, s), C.obj_scale(clip, s)
        mass = float(clip[f"obj_{s}_mass"]) if f"obj_{s}_mass" in clip else 0.0
        d[f"mano_{s}_wrist_pos"] = wrist_pos
        d[f"mano_{s}_wrist_rot"] = wrist_rot
        d[f"mano_{s}_wrist_vel"] = velocity(wrist_pos, dt, sigma)
        d[f"mano_{s}_wrist_angvel"] = angular_velocity(wrist_rot.astype(np.float64), dt, sigma)
        d[f"mano_{s}_joints"] = joints
        d[f"mano_{s}_joints_vel"] = velocity(joints, dt, sigma)
        d[f"obj_{s}_pos"] = pose[:, :3, 3].astype(np.float32)
        d[f"obj_{s}_rotmat"] = pose[:, :3, :3].astype(np.float32)
        d[f"obj_{s}_vel"] = velocity(pose[:, :3, 3], dt, sigma)
        d[f"obj_{s}_angvel"] = angular_velocity(pose[:, :3, :3], dt, sigma)
        mesh = meshes.get(obj_id, scale)
        dist, flags, local = tip_contacts(
            joints[:, tip_cols].astype(np.float64), pose, mesh["surface"], mesh["tree"], cfg.contact_threshold
        )
        d[f"tips_distance_{s}"] = dist
        d[f"contact_contact_{s}"] = flags
        d[f"contact_contact_pos_full_{s}"] = local
        meta[s] = (obj_id, scale, mass)
    return d, meta

joint_names

joint_names(robot: str, sides: tuple[str, ...]) -> list[str]

The robot MJCF's joints in qpos order: the columns of joint_pos.

Source code in eden/extensions/retargeting/flashdexretarget/pack.py

def joint_names(robot: str, sides: tuple[str, ...]) -> list[str]:
    """The robot MJCF's joints in qpos order: the columns of ``joint_pos``."""
    m = mujoco.MjModel.from_xml_path(str(robot_xml(robot, sides)))
    if m.nq != m.njnt:
        raise ValueError(f"{robot_xml(robot, sides)} has a joint with several qpos entries; joint_pos needs one each.")
    return [m.joint(j).name for j in np.argsort(m.jnt_qposadr)]

build

build(clips: Path, dataset: Path, motion: Path, fps: float, cfg, robot: str, keys: dict[str, dict]) -> None

Pack the retargeted clips of keys (clip name -> its :func:~eden.extensions.retargeting.flashdexretarget.retarget.cache_key) into motion (cfg: pack). Its object mesh directories are relative to motion's directory (../objects/<id> for <dataset>/<robot>/motion.pt).

Source code in eden/extensions/retargeting/flashdexretarget/pack.py

def build(clips: Path, dataset: Path, motion: Path, fps: float, cfg, robot: str, keys: dict[str, dict]) -> None:
    """Pack the retargeted clips of ``keys`` (clip name -> its
    :func:`~eden.extensions.retargeting.flashdexretarget.retarget.cache_key`) into ``motion`` (cfg: pack). Its object
    mesh directories are relative to ``motion``'s directory (``../objects/<id>`` for ``<dataset>/<robot>/motion.pt``).
    """
    files = _files(clips, keys)
    out = Path(motion).resolve()
    meshes = C.ObjectMeshes(dataset)
    rows, metas = [], []
    for f in files:
        d, m = load_clip(f, meshes, fps, cfg)
        rows.append(d)
        metas.append(m)
        en.logger.debug(f"[pack] {f.stem}: {len(d['joint_pos'])} frames, objects {[m[s] for s in m['sides']]}")
    sides = metas[0]["sides"]
    bad = [f.name for f, m in zip(files, metas) if m["sides"] != sides]
    if bad:
        raise ValueError(f"every clip needs the same hand sides {sides}; differs: {bad[:5]}")

    packed: dict = {k: torch.from_numpy(np.concatenate([r[k] for r in rows])) for k in rows[0]}
    frames = torch.tensor([len(r["joint_pos"]) for r in rows], dtype=torch.long)
    packed["motion_num_frames"] = frames
    packed["length_starts"] = torch.cumsum(frames, 0) - frames
    packed["motion_filename"] = [f.stem for f in files]
    for s in sides:
        packed[f"mano_{s}_joint_names"] = list(C.MANO_JOINT_NAMES)
    packed["robot"], packed["fps"] = robot, float(fps)
    packed["joint_names"] = joint_names(robot, sides)
    if len(packed["joint_names"]) != packed["joint_pos"].shape[1]:
        raise ValueError(
            f"joint_pos has {packed['joint_pos'].shape[1]} columns but {robot}'s MJCF for {sides} has "
            f"{len(packed['joint_names'])} joints."
        )

    # One object slot per distinct (right, left) object pair; a single-hand file has one per object.
    slots: dict[tuple, int] = {}
    slot_of = [slots.setdefault(tuple(m[s] for s in sides), len(slots)) for m in metas]
    for i, s in enumerate(sides):
        packed[f"{s}_object_mesh_dirs"] = [
            _relative(Path(dataset) / "objects" / key[i][0], out.parent) for key in slots
        ]
        packed[f"{s}_object_mesh_scales"] = [key[i][1] for key in slots]
        if any(key[i][2] > 0 for key in slots):
            packed[f"{s}_object_masses"] = [key[i][2] for key in slots]
        packed[f"motion_object_slot_{s}"] = torch.tensor(slot_of, dtype=torch.long)

    disks = [m["support"] for m in metas]
    if any(x is not None for x in disks):
        k = max(len(x) for x in disks if x is not None)
        table = np.zeros((len(rows), k, 5), np.float32)
        for i, x in enumerate(disks):
            if x is not None and len(x):
                table[i, : len(x)] = x
        packed["support_disks"] = torch.from_numpy(table)

    # Through a file of this process's own: an interrupted save never leaves a truncated motion.pt for the SDF step and
    # the training task to load.
    tmp = out.with_name(f"{out.name}.{os.getpid()}.tmp")
    torch.save(packed, tmp)
    os.replace(tmp, out)
    en.logger.info(f"[pack] {len(rows)} clips, {int(frames.sum())} frames, {len(slots)} object slots -> {out}")

object_meshes

object_meshes(packed: dict, root: Path) -> list[tuple[Path, float]]

Unique (visual.obj, scale) pairs a motion.pt references, in slot order; root is the motion.pt's directory, which its mesh directories are relative to.

Source code in eden/extensions/retargeting/flashdexretarget/pack.py

def object_meshes(packed: dict, root: Path) -> list[tuple[Path, float]]:
    """Unique (visual.obj, scale) pairs a motion.pt references, in slot order; ``root`` is the motion.pt's directory,
    which its mesh directories are relative to.
    """
    seen: set[tuple[str, float]] = set()
    out: list[tuple[Path, float]] = []
    for side in ("right", "left"):
        dirs = list(packed.get(f"{side}_object_mesh_dirs") or [])
        scales = list(packed.get(f"{side}_object_mesh_scales") or [1.0] * len(dirs))
        if not dirs and packed.get(f"{side}_object_mesh_dir"):
            dirs, scales = [packed[f"{side}_object_mesh_dir"]], [packed.get(f"{side}_object_mesh_scale", 1.0)]
        for rel, scale in zip(dirs, scales):
            key = (str(rel), float(scale))
            if key not in seen:
                seen.add(key)
                out.append((Path(os.path.normpath(Path(root) / str(rel) / "visual.obj")), float(scale)))
    return out

The imported-clip format (<dataset>/human_demo/<clip>.npz) and the object-mesh cache the pipeline shares.

All arrays are at the clip's fps, in a z-up world, in metres; s is right or left:

============================================ =========================== ============================================ key shape content ============================================ =========================== ============================================ fps () frame rate (one per dataset) mano_s_wrist_pos, mano_s_wrist_rot (T, 3), (T, 3, 3) MANO wrist, rotation in the robot palm's convention (see :func:mano_reference) mano_s_joints (T, 20, 3) MANO finger joints and tips, :data:MANO_JOINT_NAMES order obj_s_pose, obj_s_id (T, 4, 4), () object pose and its folder under objects/ obj_s_scale, obj_s_mass () optional mesh scale and mass ============================================ =========================== ============================================

Attributes

FINGERS module-attribute

FINGERS = ('thumb', 'index', 'middle', 'ring', 'pinky')

SURFACE_SPACING module-attribute

SURFACE_SPACING = 0.002

MANO_JOINT_NAMES module-attribute

MANO_JOINT_NAMES = tuple(sorted(f'{f}_{p}' for f in FINGERS for p in ('proximal', 'intermediate', 'distal', 'tip')))

SIDES module-attribute

SIDES = ('right', 'left')

THREAD_VARS module-attribute

THREAD_VARS = ('OMP_NUM_THREADS', 'MKL_NUM_THREADS', 'OPENBLAS_NUM_THREADS')

WORKER_MODULES module-attribute

WORKER_MODULES = ('eden.extensions.retargeting.flashdexretarget.preprocess', 'eden.extensions.retargeting.flashdexretarget.retarget')

Classes

ObjectMeshes

ObjectMeshes(pool: str | Path)

objects/<id>/visual.obj (scaled) per (object, scale): vertices, convex hull, centre of mass, and the surface points (surface, the mesh subdivided to :data:SURFACE_SPACING) with their KD-tree (tree), which measures distances to the surface.

PARAMETER	DESCRIPTION
pool	The dataset directory holding objects/. TYPE: str | Path

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def __init__(self, pool: str | Path) -> None:
    self.pool = Path(pool)
    self._cache: dict[tuple[str, float], dict] = {}

Attributes

pool instance-attribute

pool = Path(pool)

Methods:

get

get(oid: str, scale: float = 1.0) -> dict

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def get(self, oid: str, scale: float = 1.0) -> dict:
    key = (oid, float(scale))
    if key not in self._cache:
        path = self.pool / "objects" / oid / "visual.obj"
        mesh = trimesh.load(path, force="mesh", process=False, skip_materials=True)
        mesh.apply_scale(scale)
        verts = np.asarray(mesh.vertices, dtype=np.float64)
        com = mesh.center_mass if mesh.is_watertight else verts.mean(0)
        surface, _ = trimesh.remesh.subdivide_to_size(verts, mesh.faces, max_edge=SURFACE_SPACING, max_iter=30)
        self._cache[key] = {
            "verts": verts,
            "surface": surface,
            "tree": cKDTree(surface),
            "hull": np.asarray(mesh.convex_hull.vertices, dtype=np.float64),
            "com": np.asarray(com, dtype=np.float64),
        }
    return self._cache[key]

Functions:

mano_reference

mano_reference(joints: ndarray, verts: ndarray, global_rot: ndarray, side: str) -> tuple[ndarray, ndarray, ndarray]

MANO layer output -> the clip's hand arrays.

PARAMETER	DESCRIPTION
joints	(T, 16, 3) MANO joints. TYPE: ndarray
verts	(T, 778, 3) MANO vertices. TYPE: ndarray
global_rot	(T, 3, 3) MANO global orientation. TYPE: ndarray
side	right or left. TYPE: str

RETURNS	DESCRIPTION
tuple[ndarray, ndarray, ndarray]	Wrist position (T, 3), wrist rotation rebased to the robot palm frame (T, 3, 3), and the 20 named joints (T, 20, 3) in :data:MANO_JOINT_NAMES order, tips taken from :data:eden.datasets.utils.body_model.CHORD_TIP_VERTICES.

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def mano_reference(
    joints: np.ndarray, verts: np.ndarray, global_rot: np.ndarray, side: str
) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
    """MANO layer output -> the clip's hand arrays.

    Parameters
    ----------
    joints : np.ndarray
        (T, 16, 3) MANO joints.
    verts : np.ndarray
        (T, 778, 3) MANO vertices.
    global_rot : np.ndarray
        (T, 3, 3) MANO global orientation.
    side : str
        ``right`` or ``left``.

    Returns
    -------
    tuple[np.ndarray, np.ndarray, np.ndarray]
        Wrist position (T, 3), wrist rotation rebased to the robot palm frame (T, 3, 3), and the 20 named joints
        (T, 20, 3) in :data:`MANO_JOINT_NAMES` order, tips taken from
        :data:`eden.datasets.utils.body_model.CHORD_TIP_VERTICES`.
    """
    # Imported here: importing eden.datasets loads every dataset module, which only the import step needs.
    from eden.datasets.utils.body_model import CHORD_TIP_VERTICES

    first_joint = {"index": 1, "middle": 4, "pinky": 7, "ring": 10, "thumb": 13}
    wrist = joints[:, 0]
    named = {}
    for finger, j in first_joint.items():
        for k, part in enumerate(("proximal", "intermediate", "distal")):
            named[f"{finger}_{part}"] = joints[:, j + k]
        named[f"{finger}_tip"] = verts[:, CHORD_TIP_VERTICES[finger]]
    joints20 = np.stack([named[n] for n in MANO_JOINT_NAMES], axis=1)
    rebase = np.array([[-1, 0, 0], [0, 0, 1], [0, 1, 0]], dtype=np.float64)
    if side == "left":
        rebase = rebase @ np.diag([-1.0, -1.0, 1.0])
    rot = np.asarray(global_rot, dtype=np.float64) @ rebase
    return wrist.astype(np.float32), rot.astype(np.float32), joints20.astype(np.float32)

single_threaded_workers

single_threaded_workers() -> Iterator[None]

Each of :data:THREAD_VARS the caller has not set is 1 while worker processes start inside, then removed again: a fresh worker process reads them as it loads numpy (or OpenMP), so N workers keep to N cores, and the caller's own environment is left as it was.

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

@contextlib.contextmanager
def single_threaded_workers() -> Iterator[None]:
    """Each of :data:`THREAD_VARS` the caller has not set is ``1`` while worker processes start inside, then removed
    again: a fresh worker process reads them as it loads numpy (or OpenMP), so N workers keep to N cores, and the
    caller's own environment is left as it was.
    """
    added = [v for v in THREAD_VARS if v not in os.environ]
    os.environ.update(dict.fromkeys(added, "1"))
    try:
        yield
    finally:
        for v in added:
            os.environ.pop(v, None)

worker_pool

worker_pool(workers: int) -> Iterator[ProcessPoolExecutor]

workers processes for the CPU steps (preprocess, IK), each with one BLAS / OpenMP thread.

They fork from a forkserver, a fresh process that imports :data:WORKER_MODULES once under :func:single_threaded_workers, not from this one: a fork of this process would keep its thread pools as they are (numpy's sized to every core), and a spawned worker would spend seconds importing Eden. The server lives as long as this process, so the thread counts are the ones of the first pool.

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

@contextlib.contextmanager
def worker_pool(workers: int) -> Iterator[ProcessPoolExecutor]:
    """``workers`` processes for the CPU steps (preprocess, IK), each with one BLAS / OpenMP thread.

    They fork from a ``forkserver``, a fresh process that imports :data:`WORKER_MODULES` once under
    :func:`single_threaded_workers`, not from this one: a fork of this process would keep its thread pools as they
    are (numpy's sized to every core), and a spawned worker would spend seconds importing Eden. The server lives as
    long as this process, so the thread counts are the ones of the first pool.
    """
    ctx = mp.get_context("forkserver")
    ctx.set_forkserver_preload(list(WORKER_MODULES))
    with single_threaded_workers(), ProcessPoolExecutor(workers, mp_context=ctx) as ex:
        yield ex

load_clip

load_clip(path: str | Path) -> dict[str, ndarray]

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def load_clip(path: str | Path) -> dict[str, np.ndarray]:
    with np.load(path, allow_pickle=False) as z:
        return {k: z[k] for k in z.files}

save_clip

save_clip(path: str | Path, clip: dict[str, ndarray]) -> None

Write through a temporary file of this process's own, so an interrupted run never leaves a truncated clip and two runs never write one file.

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def save_clip(path: str | Path, clip: dict[str, np.ndarray]) -> None:
    """Write through a temporary file of this process's own, so an interrupted run never leaves a truncated clip and
    two runs never write one file.
    """
    path = Path(path)
    tmp = path.with_name(f"{path.stem}.{os.getpid()}.tmp.npz")
    np.savez(tmp, **clip)
    os.replace(tmp, path)

list_clips

list_clips(clip_dir: str | Path) -> list[Path]

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def list_clips(clip_dir: str | Path) -> list[Path]:
    return sorted(p for p in Path(clip_dir).glob("*.npz") if not p.name.endswith(".tmp.npz"))

sides

sides(clip: dict) -> tuple[str, ...]

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def sides(clip: dict) -> tuple[str, ...]:
    return tuple(s for s in SIDES if f"mano_{s}_wrist_pos" in clip)

num_frames

num_frames(clip: dict) -> int

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def num_frames(clip: dict) -> int:
    return len(clip[f"mano_{sides(clip)[0]}_wrist_pos"])

joints

joints(clip: dict, side: str) -> ndarray

(T, 20, 3) MANO joints in :data:MANO_JOINT_NAMES order.

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def joints(clip: dict, side: str) -> np.ndarray:
    """(T, 20, 3) MANO joints in :data:`MANO_JOINT_NAMES` order."""
    names = MANO_JOINT_NAMES
    if f"mano_{side}_joint_names" in clip:
        names = [str(n) for n in clip[f"mano_{side}_joint_names"]]
    j = clip[f"mano_{side}_joints"]
    return j[:, [names.index(n) for n in MANO_JOINT_NAMES]].astype(np.float64)

tips

tips(clip: dict, side: str) -> ndarray

(T, 5, 3) MANO fingertips, thumb to pinky.

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def tips(clip: dict, side: str) -> np.ndarray:
    """(T, 5, 3) MANO fingertips, thumb to pinky."""
    cols = [MANO_JOINT_NAMES.index(f"{f}_tip") for f in FINGERS]
    return joints(clip, side)[:, cols]

obj_id

obj_id(clip: dict, side: str) -> str

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def obj_id(clip: dict, side: str) -> str:
    return str(clip[f"obj_{side}_id"])

obj_scale

obj_scale(clip: dict, side: str) -> float

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def obj_scale(clip: dict, side: str) -> float:
    return float(clip[f"obj_{side}_scale"]) if f"obj_{side}_scale" in clip else 1.0

obj_pose

obj_pose(clip: dict, side: str) -> ndarray

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def obj_pose(clip: dict, side: str) -> np.ndarray:
    return clip[f"obj_{side}_pose"].astype(np.float64)

shared_object

shared_object(clip: dict) -> bool

Both hands on one object: the same id and the same pose track.

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def shared_object(clip: dict) -> bool:
    """Both hands on one object: the same id and the same pose track."""
    if sides(clip) != SIDES or obj_id(clip, "right") != obj_id(clip, "left"):
        return False
    return bool(np.allclose(obj_pose(clip, "right"), obj_pose(clip, "left"), atol=1e-6))

objects

objects(clip: dict) -> list[str]

The clip's distinct object bodies, as hand sides (a shared object counts once).

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def objects(clip: dict) -> list[str]:
    """The clip's distinct object bodies, as hand sides (a shared object counts once)."""
    return ["right"] if shared_object(clip) else list(sides(clip))

clip_fps

clip_fps(paths) -> float

The frame rate the clips share (every clip stores its own fps).

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def clip_fps(paths) -> float:
    """The frame rate the clips share (every clip stores its own ``fps``)."""
    rates = set()
    for p in paths:
        with np.load(p, allow_pickle=False) as z:
            if "fps" not in z.files:
                raise ValueError(f"{p.name} has no 'fps' entry; re-import it with `eden retarget flashdex import`.")
            rates.add(float(z["fps"]))
    if len(rates) != 1:
        raise ValueError(
            f"The clips of one dataset need one fps, got {sorted(rates)}; re-import them with one `fps` setting."
        )
    return rates.pop()

object_ids

object_ids(paths) -> list[str]

Source code in eden/extensions/retargeting/flashdexretarget/clips.py

def object_ids(paths) -> list[str]:
    ids = set()
    for p in paths:
        with np.load(p, allow_pickle=False) as z:
            ids.update(str(z[f"obj_{s}_id"]) for s in SIDES if f"obj_{s}_id" in z.files)
    return sorted(ids)

Reject rules over imported clips: the per-clip features they test, and the verdict.

Functions:

features

features(clip: dict, meshes: ObjectMeshes, cfg, fps: float) -> dict

Per-clip quantities the reject rules test; cfg: the filters config. Heights are of an object's lowest hull point above the lowest disk top; the floor and the disks under its centre of mass support it.

Source code in eden/extensions/retargeting/flashdexretarget/filters.py

def features(clip: dict, meshes: C.ObjectMeshes, cfg, fps: float) -> dict:
    """Per-clip quantities the reject rules test; cfg: the filters config. Heights are of an object's lowest
    hull point above the lowest disk top; the floor and the disks under its centre of mass support it.
    """
    n = cfg.edge_frames
    disks = [d for d in np.asarray(clip.get("support_disks", np.zeros((0, 5)))).reshape(-1, 5) if d[3] > 0]
    z_desk = min(d[2] for d in disks) if disks else 0.0
    T = C.num_frames(clip)
    out: dict = {"frames": T, "seconds": T / fps, "disks": len(disks)}
    acc = [0.0]
    for s in C.sides(clip):
        w = _smooth(clip[f"mano_{s}_wrist_pos"].astype(np.float64), cfg.wrist_acc_spike.smooth)
        speed = np.linalg.norm(np.diff(w, axis=0), axis=1) * fps
        if len(speed) > 1:
            acc.append(float(np.abs(np.diff(speed)).max() * fps))
    out["wrist_acc_max"] = max(acc)
    offdisk, z_first, z_last, first_sup = 0, np.inf, np.inf, 1.0
    for s in C.objects(clip):
        mesh = meshes.get(C.obj_id(clip, s), C.obj_scale(clip, s))
        pose = C.obj_pose(clip, s)
        p, com = pose[:, :3, 3], np.einsum("tij,j->ti", pose[:, :3, :3], mesh["com"]) + pose[:, :3, 3]
        bottom = _bottom(mesh["hull"], pose)
        oz, on_floor = bottom - z_desk, bottom < cfg.contact
        supported = on_floor.copy()
        for x, y, _, r, _ in disks:
            supported |= np.hypot(com[:, 0] - x, com[:, 1] - y) < r
        free = ~_touched(clip, s, meshes, cfg.contact)
        low = free & (oz < cfg.obj_offdisk_untouched.max_height)  # higher up, an untouched object is held anyway
        offdisk = max(offdisk, int((low & ~supported).sum()))
        sunk = free & ~on_floor
        if sunk[:n].any():
            z_first = min(z_first, float(oz[:n][sunk[:n]].mean()))
        if sunk[-n:].any():
            z_last = min(z_last, float(oz[-n:][sunk[-n:]].mean()))
        sup = [
            _supported(
                mesh["hull"] @ pose[t, :3, :3].T + p[t],
                com[t],
                cfg.first_unsupported.base_band,
                cfg.first_unsupported.slack,
            )
            for t in range(min(n, T))
            if low[t]
        ]
        first_sup = min(first_sup, float(np.mean(sup)) if sup else 1.0)
    out.update(
        obj_offdisk_untouched_frames=offdisk,
        obj_z_first=float(z_first),
        obj_z_last=float(z_last),
        first_supported=first_sup,
    )
    return out

reject_reasons

reject_reasons(f: dict, cfg) -> list[str]

Source code in eden/extensions/retargeting/flashdexretarget/filters.py

def reject_reasons(f: dict, cfg) -> list[str]:
    failed = {
        "obj_offdisk_untouched": f["obj_offdisk_untouched_frames"] >= cfg.obj_offdisk_untouched.frames,
        "wrist_acc_spike": f["wrist_acc_max"] > cfg.wrist_acc_spike.max_acc,
        "obj_below_desk": min(f["obj_z_first"], f["obj_z_last"]) < cfg.obj_below_desk.min_z,
        "too_short": f["seconds"] < cfg.too_short.min_sec,
        "first_unsupported": f["first_supported"] < 1.0,
    }
    return [rule for rule, bad in failed.items() if bad]

Quality flags of a retargeted clip, from a kinematic replay of its qpos (a flagged clip is still packed).

Attributes

Functions:

flags

flags(clip: dict, robot: str, pool, cfg, fps: float) -> dict

Clip carries qpos (T, nq of the robot MJCF) aligned with its other arrays; cfg: retarget.

Source code in eden/extensions/retargeting/flashdexretarget/quality.py

def flags(clip: dict, robot: str, pool, cfg, fps: float) -> dict:
    """Clip carries qpos (T, nq of the robot MJCF) aligned with its other arrays; cfg: retarget."""
    th = cfg.quality
    sides = C.sides(clip)
    objects = {s: (C.obj_id(clip, s), C.obj_scale(clip, s)) for s in sides}
    disks = np.asarray(clip.get("support_disks", np.zeros((0, 5)))).reshape(-1, 5)
    m = S.build(robot, sides, objects, pool, cfg.scene, disks, shared=C.shared_object(clip)).compile()
    d = mujoco.MjData(m)
    qpos = clip["qpos"]
    T, nq_robot = len(qpos), qpos.shape[1]
    z_desk = float(disks[:, 2].min()) if len(disks) else 0.0
    rng = np.random.default_rng(0)

    hand = {s: [] for s in sides}
    for g in range(m.ngeom):
        name = m.geom(g).name
        if name.startswith("collision_hand_") and m.geom_type[g] == mujoco.mjtGeom.mjGEOM_MESH:
            s = "left" if "left" in name else "right"
            if s in hand:
                mid = m.geom_dataid[g]
                v = m.mesh_vert[m.mesh_vertadr[mid] : m.mesh_vertadr[mid] + m.mesh_vertnum[mid]]
                hand[s].append((g, v[rng.choice(len(v), min(60, len(v)), replace=False)]))
    hand_ids = {g for g in range(m.ngeom) if m.geom(g).name.startswith("collision_hand_")}
    where = S.robot_xml(robot, sides)

    def find(objtype: mujoco.mjtObj, name: str) -> int:  # a missing name raises: -1 would read the last element
        return S.name_id(m, objtype, name, where)

    desk_ids = {find(mujoco.mjtObj.mjOBJ_GEOM, f"support_{k}") for k in range(len(disks))}
    obj_ids = {g for g in range(m.ngeom) if "_object_" in m.geom(g).name}
    tip_sites = {s: [find(mujoco.mjtObj.mjOBJ_SITE, f"{s}_{t}") for t in TIPS] for s in sides}
    mano_tips = {s: C.tips(clip, s) for s in sides}

    palm_body = {s: find(mujoco.mjtObj.mjOBJ_BODY, f"{s[0].upper()}_forearm_rot_y_link") for s in sides}
    palm = {s: np.zeros((T, 3)) for s in sides}
    tip_err, below_out, desk_pen, obj_pen = [], np.zeros(T), np.zeros(T), np.zeros(T)
    for t in range(T):
        d.qpos[:nq_robot] = qpos[t]
        for i, s in enumerate(sides):
            pose = C.obj_pose(clip, s)[t]
            q = np.zeros(4)
            mujoco.mju_mat2Quat(q, pose[:3, :3].reshape(-1))
            d.qpos[nq_robot + 7 * i : nq_robot + 7 * i + 7] = np.concatenate([pose[:3, 3], q])
        mujoco.mj_forward(m, d)
        for s in sides:
            palm[s][t] = d.xpos[palm_body[s]]
        tip_err.append(
            np.concatenate([np.linalg.norm(d.site_xpos[tip_sites[s]] - mano_tips[s][t], axis=1) for s in sides])
        )
        for c in d.contact[: d.ncon]:
            g1, g2, depth = c.geom1, c.geom2, max(0.0, -float(c.dist))
            pair = {g1, g2}
            if pair & hand_ids and pair & obj_ids:
                obj_pen[t] = max(obj_pen[t], depth)
            elif pair & hand_ids and pair & desk_ids:
                desk_pen[t] = max(desk_pen[t], depth)
        if len(disks):
            pts = np.concatenate(
                [v @ d.geom_xmat[g].reshape(3, 3).T + d.geom_xpos[g] for s in sides for g, v in hand[s]]
            )
            on = np.zeros(len(pts), dtype=bool)
            for x, y, _, r, _ in disks:
                on |= np.hypot(pts[:, 0] - x, pts[:, 1] - y) < r + 0.02
            if (~on).any():
                below_out[t] = max(0.0, float((z_desk - pts[~on, 2]).max()))
    tip_err = np.asarray(tip_err)
    acc = [0.0]
    for s in sides:
        v = np.linalg.norm(np.diff(palm[s], axis=0), axis=1) * fps
        if len(v) > 1:
            acc.append(float(np.abs(np.diff(v)).max() * fps))
    out = {
        "palm_acc_max": max(acc),
        "tip_err_median_m": float(np.median(tip_err)),
        "tip_far_frac": float((tip_err > th.tip_tracking.max_dist).mean()),
        "hand_below_desk_outside_max_m": float(below_out.max()),
        "hand_obj_pen_max_m": float(obj_pen.max()),
        "hand_desk_pen_frames": int(
            (desk_pen[th.hand_desk_penetration.after :] > th.hand_desk_penetration.max_depth).sum()
        ),
    }
    flagged = {
        "palm_acc_spike": out["palm_acc_max"] > th.palm_acc_spike.max_acc,
        "hand_far_below_desk": out["hand_below_desk_outside_max_m"] > th.hand_far_below_desk.max_depth,
        "tip_tracking": out["tip_far_frac"] > th.tip_tracking.max_frac,
        "hand_obj_penetration": out["hand_obj_pen_max_m"] > th.hand_obj_penetration.max_depth,
        "hand_desk_penetration": out["hand_desk_pen_frames"] >= th.hand_desk_penetration.frames,
    }
    out["flags"] = ";".join(name for name, bad in flagged.items() if bad)
    return out

See also

• Retargeting guide
• Dex retargeting guide
• Datasets