Pose Estimation

Pose estimators recover human motion from monocular video as an AMASS-style SMPL-X clip that the retargeting extensions consume. An estimator is a PoseEstimatorBase subclass configured by PoseEstimatorOptions; the backend that ships is GVHMR, installed with the pose extra (pip install 'eden-world[pose]', or uv sync --inexact --extra pose in a checkout) and imported lazily. The pose estimation guide covers eden pose gvhmr.

from eden.extensions.pose_estimation.base import PoseEstimatorBase
from eden.options.extensions.pose_estimation import PoseEstimatorOptions

Base class and options

Base class for pose-estimation backends (monocular video → human motion).

Classes

PoseEstimatorBase

PoseEstimatorBase(options: T)

Bases: ConfigurableMixin[PoseEstimatorOptions]

Abstract base class for recovering human motion from video.

Takes no env, unlike the retargeting and grasp-sampling bases: estimation is offline preprocessing whose output is a motion clip, and the robot only enters at retargeting.

Backends expose the estimate at two levels — :meth:estimate keeps the backend's own frames and layout (so camera-space work and provenance survive), :meth:estimate_clip packs it into the AMASS-style Z-up SMPL clip the retargeting extensions consume.

No registry, as in retargeting: backends are chosen by CLI subcommand, never resolved by name from a config.

Source code in eden/utils/common.py

def __init__(self, options: T) -> None:
    self._uid = gs.UID()
    self._options: T = options

    mirror_options_onto(self, self._options)

    # Deferred, not module-scope — see the NOTE above this module's imports.
    from eden.options._metadata import OPTION_METADATA_KEYS

    # check if there are unused parameters in the options. ``_extra_option_params`` are legitimate
    # options-only fields (not mirrored as attrs), so they are expected to be "unused" here.
    unused_params = set(self._options.keys()) - set(self.get_parameter_names())
    tolerated = {"name", "class_name"} | set(OPTION_METADATA_KEYS) | self._extra_option_params
    # ``params`` is a declared framework field on function-capable term options. An *empty* one is
    # legitimately unused when the same options class configures a class-based term; a non-empty
    # ``params`` on a class-based term would be silently ignored, so it stays invalid.
    if not getattr(self._options, "params", None):
        tolerated.add("params")
    invalid_params = unused_params - tolerated
    if invalid_params:
        # Name the term, not only the class: for a function term the class is the shared
        # ``*FuncWrapper``, which cannot tell the author which of a dozen terms is wrong.
        # Same wording as the construction-time check (``ManagerTermOptions._validate_param_names``).
        term_name = getattr(self._options, "name", None)
        subject = f"term '{term_name}' ({type(self).__name__})" if term_name else type(self).__name__
        raise ValueError(
            invalid_parameters_message(subject, invalid_params, self.get_configure_specs(), self._retired_params)
        )

Attributes

checkpoint_root class-attribute instance-attribute

checkpoint_root: str = ''

ckpt_path class-attribute instance-attribute

ckpt_path: str = ''

output_root class-attribute instance-attribute

output_root: str = ''

motion_fps class-attribute instance-attribute

motion_fps: float = 30.0

verbose class-attribute instance-attribute

verbose: bool = False

Methods:

estimate abstractmethod

estimate(video: str | Path, **overrides) -> dict

Recover the motion in video, in the backend's own frames and layout.

Source code in eden/extensions/pose_estimation/base.py

@abstractmethod
def estimate(self, video: str | Path, **overrides) -> dict:
    """Recover the motion in ``video``, in the backend's own frames and layout."""
    raise NotImplementedError

estimate_clip abstractmethod

estimate_clip(video: str | Path, **overrides) -> dict[str, ndarray]

Recover video as an AMASS-style, Z-up SMPL clip ready for np.savez.

Source code in eden/extensions/pose_estimation/base.py

@abstractmethod
def estimate_clip(self, video: str | Path, **overrides) -> dict[str, np.ndarray]:
    """Recover ``video`` as an AMASS-style, Z-up SMPL clip ready for ``np.savez``."""
    raise NotImplementedError

close

close() -> None

Release whatever the backend holds (a loaded model, GPU memory); no-op if nothing.

Always safe to call, so callers can use try/finally regardless of backend.

Source code in eden/extensions/pose_estimation/base.py

def close(self) -> None:
    """Release whatever the backend holds (a loaded model, GPU memory); no-op if nothing.

    Always safe to call, so callers can use ``try``/``finally`` regardless of backend.
    """

Configuration options for pose-estimation extensions (video → human motion).

:class:PoseEstimatorOptions mirrors :class:~eden.extensions.pose_estimation.base.PoseEstimatorBase's attributes, so the shared fields are type-checked at configure() time. Backend-specific knobs stay as class attributes on the backend and arrive as extras — e.g. GVHMR's static_cam / use_dpvo / f_mm.

Classes

PoseEstimatorOptions

PoseEstimatorOptions(**data)

Bases: DynamicConfigurableOptions

Options shared by every pose-estimation backend.

PARAMETER	DESCRIPTION
checkpoint_root	Directory of externally downloaded weights. Empty keeps the backend's own resolution (GVHMR: $GVHMR_CHECKPOINT_ROOT, else ./inputs/checkpoints). TYPE: str
ckpt_path	Model checkpoint file. Empty derives it from checkpoint_root. TYPE: str
output_root	Where the backend writes — and reuses, which is what makes a re-run cheap — its per-video intermediates. Empty keeps the backend default, usually relative to cwd. TYPE: str
motion_fps	Frame rate stamped on the clip: the rate the output is expressed in, not necessarily the source video's. GVHMR relabels every input to 30 fps without resampling frames and integrates root motion at that rate, so 30 is right there even for a 60 fps source. TYPE: float
verbose	Save the backend's intermediate preprocessing overlays. TYPE: bool

Source code in eden/options/options.py

def __init__(self, **data):
    # Seed type metadata as construction data so BaseModel validates it into
    # __pydantic_extra__ (no-op for extra="forbid" classes, which use
    # the serialization envelope instead).
    data.update(option_metadata(self))
    # NOTE: all of the option modifications/validations should be done in the __init__ method
    try:
        BaseModel.__init__(self, **data)
    except ValidationError as exc:
        # A misspelled field on an extra="forbid" class should teach the field set, not just
        # name the typo. Only rewritten when every error is extra_forbidden.
        enriched = _enrich_extra_forbidden(type(self), exc)
        if enriched is None:
            raise
        raise enriched from None

Attributes

checkpoint_root class-attribute instance-attribute

checkpoint_root: str = ''

ckpt_path class-attribute instance-attribute

ckpt_path: str = ''

output_root class-attribute instance-attribute

output_root: str = ''

motion_fps class-attribute instance-attribute

motion_fps: float = 30.0

verbose class-attribute instance-attribute

verbose: bool = False

GVHMR

Eden wrapper around GVHMR (world-grounded human motion recovery from monocular video).

GVHMR (zju3dv/GVHMR <https://github.com/zju3dv/GVHMR>, SIGGRAPH Asia 2024) is consumed through the packaged fork Kashu7100/eden_gvhmr <https://github.com/Kashu7100/eden_gvhmr> (import name hmr4d). :meth:GvhmrEstimator.estimate_clip writes what eden retarget soma --body-model smplx reads, so video → clip → robot qpos is two steps. See docs/guides/pose-estimation.md for setup.

In-process. The fork installs beside Eden: it drops upstream's pytorch3d requirement for inference, relaxes a stale numpy<1.24 cap, does not compose Hydra configs (so it does not fight a host that owns Hydra's global state), and carries no VCS dependency, so it ships on PyPI. pip install 'eden-world[pose]' resolves additively into Eden's environment and this class simply imports hmr4d.

One consequence worth knowing: GVHMR runs on Eden's video decoder and person detector rather than the versions its numerics were tuned against. Measured one change at a time on the bundled tennis clip, av 13 → 17 moves body joints by 3.6° max and ultralytics 8.2.42 → 8.4.113 moves the detected boxes by 8.5 px and body joints by 3.4°; numpy (including the 1.x/2.x jump), timm and imageio are all bit-identical. The estimate stays valid — it is just not bit-comparable with the published checkpoint's reference stack.

Classes

GvhmrEstimator

GvhmrEstimator(options: PoseEstimatorOptions)

Bases: PoseEstimatorBase

Monocular video → world-grounded SMPL-X motion.

The shared options (checkpoint_root, ckpt_path, output_root, motion_fps, verbose) are documented on :class:~eden.options.extensions.pose_estimation.PoseEstimatorOptions; the fields below are GVHMR's own and reach the options as extras.

The model is loaded once, on the first :meth:estimate call, and reused across videos. GVHMR is CUDA-only; select a GPU with CUDA_VISIBLE_DEVICES.

No overlay rendering. GVHMR can draw in-camera / world overlay videos, but that path needs two dependencies inference does not — pytorch3d (built against your torch/CUDA) and chumpy from git — neither of which can be declared, so the pose extra does not install them. Rather than offer a flag that fails after the whole pipeline has run, Eden does not expose rendering at all: the product here is the .npz. To eyeball an estimate, install those two and use GVHMR's own CLI (python -m hmr4d.demo). verbose still works — its bbox / keypoint overlays are drawn with cv2 and need nothing extra.

PARAMETER	DESCRIPTION
static_cam	Skip visual odometry (assume the camera does not move). Much faster, and the right choice for tripod footage; wrong for handheld video, where camera motion would be absorbed into the body motion. TYPE: bool
use_dpvo	Use DPVO instead of the default SimpleVO for camera motion. Requires GVHMR's dpvo extra and its checkpoint. TYPE: bool
f_mm	Full-frame-equivalent focal length in mm. None lets GVHMR estimate it. TYPE: float | None

Source code in eden/extensions/pose_estimation/gvhmr/estimator.py

def __init__(self, options: PoseEstimatorOptions) -> None:
    # Set before anything that can raise, so close() is safe even on a failed construction.
    self._model = None
    self._skeleton_model = None
    super().__init__(options)
    if abs(self.motion_fps - _ASSUMED_VIDEO_FPS) > _FPS_TOLERANCE:
        raise ValueError(
            f"motion_fps={self.motion_fps:g} but GVHMR always expresses its output at "
            f"{_ASSUMED_VIDEO_FPS:g} fps — it relabels every input to that rate and integrates root motion "
            f"at it. Stamping any other rate mislabels the clip without changing the motion, which is worse "
            f"than the default. Re-encode the source video instead if you need a different rate."
        )

Attributes

static_cam class-attribute instance-attribute

static_cam: bool = False

use_dpvo class-attribute instance-attribute

use_dpvo: bool = False

f_mm class-attribute instance-attribute

f_mm: float | None = None

Methods:

close

close() -> None

Drop the loaded model and free its GPU memory (idempotent).

Source code in eden/extensions/pose_estimation/gvhmr/estimator.py

def close(self) -> None:
    """Drop the loaded model and free its GPU memory (idempotent)."""
    self._model = None
    self._skeleton_model = None
    try:
        import torch

        if torch.cuda.is_available():
            torch.cuda.empty_cache()
    except ImportError:  # torch is always present in Eden; be safe during interpreter teardown
        pass

estimate

estimate(video: str | Path, **overrides) -> dict

Recover the motion in video, verbatim in GVHMR's own frames.

**overrides take any of :data:_RECOVER_KEYS for this call only.

RETURNS	DESCRIPTION
dict	global — SMPL-X params in GVHMR's gravity-aligned Y-up world (global_orient (T, 3), body_pose (T, 63), betas (T, 10), transl (T, 3)); incam — the same in the camera frame; pelvis_j0 (T, 3); K_fullimg; output_dir; video_fps (the source rate). :meth:estimate_clip returns the Z-up, AMASS-style form Eden reads.

Source code in eden/extensions/pose_estimation/gvhmr/estimator.py

def estimate(self, video: str | Path, **overrides) -> dict:
    """Recover the motion in ``video``, verbatim in GVHMR's own frames.

    ``**overrides`` take any of :data:`_RECOVER_KEYS` for this call only.

    Returns
    -------
    dict
        ``global`` — SMPL-X params in GVHMR's gravity-aligned **Y-up** world
        (``global_orient (T, 3)``, ``body_pose (T, 63)``, ``betas (T, 10)``,
        ``transl (T, 3)``); ``incam`` — the same in the camera frame; ``pelvis_j0
        (T, 3)``; ``K_fullimg``; ``output_dir``; ``video_fps`` (the source rate).
        :meth:`estimate_clip` returns the Z-up, AMASS-style form Eden reads.
    """
    path = Path(video).expanduser()
    if not path.is_file():
        raise FileNotFoundError(f"Video does not exist: {path}")
    unknown = set(overrides) - set(_RECOVER_KEYS)
    if unknown:
        raise TypeError(f"Unknown estimate() override(s) {sorted(unknown)}; expected {list(_RECOVER_KEYS)}.")

    kwargs = {key: getattr(self, key) for key in _RECOVER_KEYS}
    kwargs.update(overrides)
    kwargs["output_root"] = kwargs["output_root"] or None  # GVHMR keys its defaults on None

    video_path = str(path.resolve())
    with _checkpoint_root_env(self.checkpoint_root):
        # Load first: a missing backend must surface as _require_hmr4d's actionable
        # ImportError, not as `av` failing to import inside _video_fps.
        model = self._ensure_model()

        # Vet the source rate up front. Warning only after `recover()` would mean spending the
        # whole pipeline before telling the caller the clip they get is time-warped.
        fps = _video_fps(video_path)
        if fps is not None and abs(fps - _ASSUMED_VIDEO_FPS) > _FPS_TOLERANCE:
            en.logger.warning(
                f"{path.name} is {fps:g} fps, but GVHMR relabels every input to {_ASSUMED_VIDEO_FPS:g} fps "
                f"without resampling and integrates root motion at that rate — the recovered trajectory is "
                f"time-warped by {fps / _ASSUMED_VIDEO_FPS:.2f}x. Re-encode the video to "
                f"{_ASSUMED_VIDEO_FPS:g} fps for a physically correct clip."
            )

        out = model.recover(video_path, **kwargs)
        pelvis_j0 = self._pelvis_rest_offset(out["smpl_params_global"]["betas"])

    # Plain numpy on the way out: the caller should not have to care that GVHMR returns
    # CUDA tensors, and `convert` takes numpy so a saved clip stays framework-agnostic.
    return {
        "global": {key: _to_numpy(value) for key, value in out["smpl_params_global"].items()},
        "incam": {key: _to_numpy(value) for key, value in out["smpl_params_incam"].items()},
        "pelvis_j0": pelvis_j0,
        "K_fullimg": _to_numpy(out["K_fullimg"]),
        "output_dir": str(out["output_dir"]),
        "video_fps": fps,
    }

estimate_clip

estimate_clip(video: str | Path, **overrides) -> dict[str, ndarray]

Recover video as an AMASS-style, Z-up SMPL-X clip.

Ready for :func:~eden.extensions.pose_estimation.gvhmr.convert.save_amass_npz; :func:~eden.extensions.pose_estimation.gvhmr.convert.to_amass_smplx_clip lists the fields.

Source code in eden/extensions/pose_estimation/gvhmr/estimator.py

def estimate_clip(self, video: str | Path, **overrides) -> dict[str, np.ndarray]:
    """Recover ``video`` as an AMASS-style, **Z-up** SMPL-X clip.

    Ready for :func:`~eden.extensions.pose_estimation.gvhmr.convert.save_amass_npz`;
    :func:`~eden.extensions.pose_estimation.gvhmr.convert.to_amass_smplx_clip` lists the fields.
    """
    return to_amass_smplx_clip(self.estimate(video, **overrides), motion_fps=self.motion_fps, source_video=video)

Functions:

Conversion to an AMASS-style clip

GVHMR's world-frame output is Y-up; these helpers rotate it to Eden's Z-up and write an AMASS-style SMPL-X .npz.

GVHMR's world-frame SMPL-X output → an AMASS-style, Z-up SMPL-X clip.

Two things change. Up axis: GVHMR predicts in a gravity-aligned frame whose +Y is up (its "ay" frame — hmr4d.utils.geo.hmr_global's "az"->"ay" is -90° about X), while Eden/Genesis/AMASS are Z-up. Pose layout: GVHMR returns global_orient (T, 3) + body_pose (T, 63); AMASS packs one poses (T, 165), whose jaw/eyes/hands tail stays zero because GVHMR does not estimate it.

Does not import GVHMR, so the conversion is testable without the backend installed.

Attributes

SMPLX_POSE_DIM module-attribute

SMPLX_POSE_DIM = 165

ROOT_ORIENT_SLICE module-attribute

ROOT_ORIENT_SLICE = slice(0, 3)

BODY_POSE_SLICE module-attribute

BODY_POSE_SLICE = slice(3, 66)

Y_UP_TO_Z_UP_AXIS_ANGLE module-attribute

Y_UP_TO_Z_UP_AXIS_ANGLE = (0.5 * np.pi, 0.0, 0.0)

Functions:

y_up_to_z_up_quat

y_up_to_z_up_quat() -> Tensor

The wxyz quaternion of :data:Y_UP_TO_Z_UP_AXIS_ANGLE (maps +Y to +Z).

Source code in eden/extensions/pose_estimation/gvhmr/convert.py

def y_up_to_z_up_quat() -> torch.Tensor:
    """The wxyz quaternion of :data:`Y_UP_TO_Z_UP_AXIS_ANGLE` (maps ``+Y`` to ``+Z``)."""
    return quat_from_axis_angle(torch.tensor(Y_UP_TO_Z_UP_AXIS_ANGLE, dtype=torch.float32))

y_up_to_z_up_root

y_up_to_z_up_root(global_orient: ndarray, transl: ndarray, pelvis_j0: ndarray) -> tuple[ndarray, ndarray]

Rotate an SMPL-X root (global_orient, transl) pair from Y-up into Z-up.

body_pose is unaffected — SMPL joint rotations are parent-relative, so only the root sees a change of world frame.

The translation is not simply R @ transl: SMPL-X rotates the body about the rest-pose pelvis J0(betas) (BodyModelSMPLX.get_skeleton(betas)[..., 0, :]), which puts the pelvis at J0 + transl in the world, so rotating about the origin needs::

transl' = R @ (transl + J0) - J0

global_orient / transl are (T, 3), pelvis_j0 (T, 3) or (3,); returns the Z-up pair as float32.

Source code in eden/extensions/pose_estimation/gvhmr/convert.py

def y_up_to_z_up_root(
    global_orient: np.ndarray, transl: np.ndarray, pelvis_j0: np.ndarray
) -> tuple[np.ndarray, np.ndarray]:
    """Rotate an SMPL-X root ``(global_orient, transl)`` pair from Y-up into Z-up.

    ``body_pose`` is unaffected — SMPL joint rotations are parent-relative, so only the root
    sees a change of world frame.

    The translation is **not** simply ``R @ transl``: SMPL-X rotates the body about the
    rest-pose pelvis ``J0(betas)`` (``BodyModelSMPLX.get_skeleton(betas)[..., 0, :]``), which
    puts the pelvis at ``J0 + transl`` in the world, so rotating about the origin needs::

        transl' = R @ (transl + J0) - J0

    ``global_orient`` / ``transl`` are ``(T, 3)``, ``pelvis_j0`` ``(T, 3)`` or ``(3,)``;
    returns the Z-up pair as float32.
    """
    orient = torch.as_tensor(np.asarray(global_orient), dtype=torch.float32)
    trans = torch.as_tensor(np.asarray(transl), dtype=torch.float32)
    if orient.ndim != 2 or orient.shape[-1] != 3 or trans.shape != orient.shape:
        raise ValueError(
            f"Expected per-frame (T, 3) arrays; got global_orient {tuple(orient.shape)}, transl {tuple(trans.shape)}."
        )
    j0 = torch.as_tensor(np.asarray(pelvis_j0), dtype=torch.float32).expand_as(trans)
    quat = y_up_to_z_up_quat()

    orient_z = axis_angle_from_quat(quat_mul(quat, quat_from_axis_angle(orient)))
    trans_z = transform_by_quat(trans + j0, quat) - j0
    return orient_z.numpy(), trans_z.numpy()

to_amass_smplx_clip

to_amass_smplx_clip(result: dict, *, motion_fps: float, source_video: str | Path | None = None) -> dict[str, ndarray]

Pack one :meth:GvhmrEstimator.estimate result into AMASS-style SMPL-X npz fields.

Emits the AMASS fields Eden's retargeting path reads (poses, trans, betas, gender, mocap_frame_rate, surface_model_type) plus gvhmr_* copies of the raw Y-up estimate, so a saved clip stays traceable to what the model predicted.

betas is averaged over the clip — AMASS carries one shape vector per sequence, and GVHMR's per-frame betas are already a clip-average repeated per frame.

Source code in eden/extensions/pose_estimation/gvhmr/convert.py

def to_amass_smplx_clip(
    result: dict, *, motion_fps: float, source_video: str | Path | None = None
) -> dict[str, np.ndarray]:
    """Pack one :meth:`GvhmrEstimator.estimate` result into AMASS-style SMPL-X npz fields.

    Emits the AMASS fields Eden's retargeting path reads (``poses``, ``trans``, ``betas``,
    ``gender``, ``mocap_frame_rate``, ``surface_model_type``) plus ``gvhmr_*`` copies of the
    raw Y-up estimate, so a saved clip stays traceable to what the model predicted.

    ``betas`` is averaged over the clip — AMASS carries one shape vector per sequence, and
    GVHMR's per-frame betas are already a clip-average repeated per frame.
    """
    _require_keys(result, ("global", "pelvis_j0", "K_fullimg"), "GVHMR result")
    g = result["global"]
    _require_keys(g, ("global_orient", "body_pose", "betas", "transl"), "GVHMR result['global']")

    orient, body_pose, betas, transl = g["global_orient"], g["body_pose"], g["betas"], g["transl"]
    pelvis_j0 = result["pelvis_j0"]
    num_frames = orient.shape[0]
    for name, (array, shape) in {
        "global_orient": (orient, (num_frames, 3)),
        "body_pose": (body_pose, (num_frames, 63)),
        "betas": (betas, (num_frames, 10)),
        "transl": (transl, (num_frames, 3)),
        "pelvis_j0": (pelvis_j0, (num_frames, 3)),
    }.items():
        if array.shape != shape:
            raise ValueError(f"GVHMR result field {name!r} has shape {array.shape}, expected {shape}.")

    orient_z, trans_z = y_up_to_z_up_root(orient, transl, pelvis_j0)
    poses = np.zeros((num_frames, SMPLX_POSE_DIM), dtype=np.float32)
    poses[:, ROOT_ORIENT_SLICE] = orient_z
    poses[:, BODY_POSE_SLICE] = body_pose

    payload = {
        "poses": poses,
        "trans": trans_z,
        "betas": betas.mean(axis=0).astype(np.float32),
        "gender": np.array("neutral"),  # GVHMR predicts with the SMPL-X *neutral* model
        "mocap_frame_rate": np.float32(motion_fps),
        "surface_model_type": np.array("smplx"),
        "gvhmr_global_orient": orient,
        "gvhmr_transl": transl,
        "gvhmr_betas": betas,
        "gvhmr_pelvis_j0": pelvis_j0,
        "gvhmr_K_fullimg": result["K_fullimg"],
    }
    if result.get("video_fps") is not None:
        payload["gvhmr_video_fps"] = np.float32(result["video_fps"])
    if source_video is not None:
        payload["source_video"] = np.array(str(source_video))
    return payload

save_amass_npz

save_amass_npz(payload: dict[str, ndarray], path: str | Path) -> Path

Write a :func:to_amass_smplx_clip payload to path with an .npz suffix (parents created).

Source code in eden/extensions/pose_estimation/gvhmr/convert.py

def save_amass_npz(payload: dict[str, np.ndarray], path: str | Path) -> Path:
    """Write a :func:`to_amass_smplx_clip` payload to ``path`` with an ``.npz`` suffix (parents created)."""
    out = Path(path).with_suffix(".npz")
    out.parent.mkdir(parents=True, exist_ok=True)
    np.savez(out, **payload)
    return out

See also

• Pose estimation guide
• Retargeting