Skip to content

Rendering Utilities

The batched Gaussian-splat renderer behind SplatCamera and BatchSplatCamera (gsplat backend), its .ply loading and conversion, and the shadow transfer that darkens a composited background where the simulation casts a shadow. The sensors and rendering guide explains when each is used.

Batched splat renderer

Batched Gaussian-splat renderer for Eden, backed by gsplat.

Two rendering regimes, selected by per_env_gaussians:

  • shared (per_env_gaussians=False) — one Gaussian set is shared by every environment and rasterized from the per-env camera poses (the env axis is used as gsplat's camera batch). The Gaussian geometry (means/quats/scales) is stored once — never replicated or transformed per env — so this is the mode for a scene that is static across envs (a photorealistic background composited under Madrona-rendered robots/objects), and it is the memory-light choice. Given a single camera pose it renders exactly one view, which the caller uses for every env (:class:~eden.entities.camera.SplatCamera).

  • per-env (per_env_gaussians=True) — each env carries its own Gaussians, whose dynamic points are rigidly driven by that env's per-body sim poses (:meth:update), then rasterized with the env axis as gsplat's batch dimension. Means and quats are materialized num_envs times, so this fits tens–hundreds of envs. Use when splat objects move independently per env, or the scene is domain-randomized per env.

Memory note: gsplat evaluates view-dependent spherical harmonics per rendered view, so peak VRAM scales with (views_per_chunk * G * K) in both modes (K = (sh_degree+1)**2). minibatch chunks the env/view axis to bound that peak — set it whenever num_envs is large; shared mode's win is avoiding the per-env geometry copy, not escaping the per-view SH cost.

The renderer is deliberately sim-agnostic: it consumes camera extrinsics and per-body poses as plain torch tensors. Eden's camera/env layer is responsible for reading those out of Genesis (see SplatComposite in :mod:eden.entities.camera_overlays).

Classes

SplatEntry dataclass 

SplatEntry(ply_path: str, pos: tuple[float, float, float] = (0.0, 0.0, 0.0), quat: tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0), scale: float = 1.0, body_row: int | None = None)

One splat PLY plus its placement and (optional) rigid-body binding.

PARAMETER DESCRIPTION
ply_path

Path to a standard 3DGS/2DGS .ply.

TYPE: str

pos

Baked into the loaded Gaussians. For a static entry this is the world placement; for a dynamic entry (body_row set) it is an extra local offset applied before the per-env body pose.

TYPE: placement DEFAULT: (0.0, 0.0, 0.0)

quat

Baked into the loaded Gaussians. For a static entry this is the world placement; for a dynamic entry (body_row set) it is an extra local offset applied before the per-env body pose.

TYPE: placement DEFAULT: (0.0, 0.0, 0.0)

scale

Baked into the loaded Gaussians. For a static entry this is the world placement; for a dynamic entry (body_row set) it is an extra local offset applied before the per-env body pose.

TYPE: placement DEFAULT: (0.0, 0.0, 0.0)

body_row

None marks a static entry (rendered identically in every env). An integer binds the entry to a rigid body: in per-env mode its Gaussians follow body_pos[:, body_row] / body_quat[:, body_row] supplied to :meth:update.

TYPE: int | None DEFAULT: None

Attributes

ply_path instance-attribute 
ply_path: str
pos class-attribute instance-attribute 
pos: tuple[float, float, float] = (0.0, 0.0, 0.0)
quat class-attribute instance-attribute 
quat: tuple[float, float, float, float] = (1.0, 0.0, 0.0, 0.0)
scale class-attribute instance-attribute 
scale: float = 1.0
body_row class-attribute instance-attribute 
body_row: int | None = None

Methods:

BatchSplatRenderer 

BatchSplatRenderer(entries: list[SplatEntry], num_envs: int, *, per_env_gaussians: bool = False, device: device | str = 'cuda', minibatch: int | None = None)

Batched gsplat renderer with a shared / per-env Gaussian switch.

Source code in eden/utils/batch_splat_renderer/renderer.py 
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
def __init__(
    self,
    entries: list[SplatEntry],
    num_envs: int,
    *,
    per_env_gaussians: bool = False,
    device: torch.device | str = "cuda",
    minibatch: int | None = None,
) -> None:
    if not entries:
        raise ValueError("BatchSplatRenderer requires at least one SplatEntry.")
    self.num_envs = num_envs
    self.per_env_gaussians = per_env_gaussians
    self.device = torch.device(device)
    self.minibatch = minibatch

    self._loaded = self._load_and_bake(entries)

    n_dyn = int(self._loaded.dynamic_mask.sum())
    if not per_env_gaussians and n_dyn:
        en.logger.warning(
            f"BatchSplatRenderer: shared mode ignores per-env motion, but "
            f"{n_dyn} Gaussian(s) are bound to rigid bodies; they render frozen at "
            f"their loaded pose. Set per_env_gaussians=True to animate them."
        )
    # Cached per-env Gaussians for per-env mode (refreshed by update()).
    self._gsb: GaussianBatchData | None = None

Attributes

num_envs instance-attribute 
num_envs = num_envs
per_env_gaussians instance-attribute 
per_env_gaussians = per_env_gaussians
device instance-attribute 
device = torch.device(device)
minibatch instance-attribute 
minibatch = minibatch
world_xyz property 
world_xyz: ndarray

World-frame xyz of every loaded Gaussian (baked placement), for debug-point drawing.

n_bodies property 
n_bodies: int

Methods:

update 
update(body_pos: Tensor, body_quat: Tensor, *, scalar_first: bool = True) -> None

Refresh per-env Gaussians from rigid-body poses (per-env mode).

PARAMETER DESCRIPTION
body_pos

(num_envs, n_bodies, 3) world positions.

TYPE: Tensor

body_quat

(num_envs, n_bodies, 4) orientations. scalar_first selects wxyz (default, Genesis) vs xyzw.

TYPE: Tensor

scalar_first

Whether body_quat is wxyz (True) or xyzw (False).

TYPE: bool DEFAULT: True

Source code in eden/utils/batch_splat_renderer/renderer.py 
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
def update(self, body_pos: Tensor, body_quat: Tensor, *, scalar_first: bool = True) -> None:
    """Refresh per-env Gaussians from rigid-body poses (per-env mode).

    Parameters
    ----------
    body_pos : Tensor
        ``(num_envs, n_bodies, 3)`` world positions.
    body_quat : Tensor
        ``(num_envs, n_bodies, 4)`` orientations. ``scalar_first`` selects
        wxyz (default, Genesis) vs xyzw.
    scalar_first : bool
        Whether ``body_quat`` is wxyz (True) or xyzw (False).
    """
    if not self.per_env_gaussians:
        return  # shared mode: nothing to animate
    body_pos = torch.as_tensor(body_pos, dtype=torch.float32, device=self.device)
    body_quat = torch.as_tensor(body_quat, dtype=torch.float32, device=self.device)
    self._gsb = batch_update_gaussians(
        self._loaded.template,
        body_pos,
        body_quat,
        point_to_body_idx=self._loaded.point_to_body_idx,
        dynamic_mask=self._loaded.dynamic_mask,
        scalar_first=scalar_first,
    )
render 
render(cam_pos: Tensor, cam_rot: Tensor, height: int, width: int, fovy: float | ndarray, *, bg_imgs: Tensor | None = None, return_alpha: bool = False, clamp: bool = True) -> tuple[Tensor, ...]

Render every env's view.

Returns (rgb, depth) or (rgb, depth, alpha) if return_alpha. rgb is (num_envs, H, W, 3) float in [0, 1] (if clamp), depth and alpha are (num_envs, H, W, 1).

Source code in eden/utils/batch_splat_renderer/renderer.py 
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
def render(
    self,
    cam_pos: Tensor,  # (num_envs, 3) world
    cam_rot: Tensor,  # (num_envs, 3, 3) world camera rotation (Genesis/OpenGL frame)
    height: int,
    width: int,
    fovy: float | np.ndarray,  # vertical FOV, degrees
    *,
    bg_imgs: Tensor | None = None,  # (num_envs, H, W, 3)
    return_alpha: bool = False,
    clamp: bool = True,
) -> tuple[Tensor, ...]:
    """Render every env's view.

    Returns ``(rgb, depth)`` or ``(rgb, depth, alpha)`` if ``return_alpha``.
    ``rgb`` is ``(num_envs, H, W, 3)`` float in ``[0, 1]`` (if ``clamp``),
    ``depth`` and ``alpha`` are ``(num_envs, H, W, 1)``.
    """
    cam_pos = torch.as_tensor(cam_pos, dtype=torch.float32, device=self.device).reshape(-1, 3)
    cam_rot = torch.as_tensor(cam_rot, dtype=torch.float32, device=self.device).reshape(-1, 3, 3)
    n = cam_pos.shape[0]
    fovy_arr = np.broadcast_to(np.asarray(fovy, dtype=np.float32), (n,)).copy()

    if self.per_env_gaussians:
        if self._gsb is None:
            # No update() yet (e.g. no dynamic bodies): expand the template once.
            self.update(
                torch.zeros(self.num_envs, max(self.n_bodies, 1), 3, device=self.device),
                torch.tensor([1.0, 0, 0, 0], device=self.device).expand(self.num_envs, max(self.n_bodies, 1), 4),
            )
        if n != self._gsb.batch_size:
            # gsplat batches means and viewmats on the same leading axis; a mismatch would
            # otherwise surface as a cryptic gsplat shape error. Broadcast a shared camera
            # to num_envs (SplatComposite does this) before calling render() in per-env mode.
            raise ValueError(
                f"per-env render expects one camera pose per env: got {n} camera pose(s) for "
                f"{self._gsb.batch_size} envs."
            )
        out = batch_env_render(
            self._gsb,
            cam_pos.reshape(n, 1, 3),
            cam_rot.reshape(n, 1, 9),
            height,
            width,
            fovy_arr.reshape(n, 1),
            bg_imgs=bg_imgs.reshape(n, 1, height, width, 3) if bg_imgs is not None else None,
            minibatch=self.minibatch,
            return_alpha=return_alpha,
        )
        # drop the singleton camera axis
        out = tuple(t.reshape(n, height, width, -1) for t in out)
    else:
        # Shared geometry: env axis is gsplat's camera batch. Chunk it so the
        # per-view SH evaluation stays within VRAM (see module memory note).
        out = self._render_shared_chunked(cam_pos, cam_rot, height, width, fovy_arr, bg_imgs, return_alpha)

    if clamp:
        out = (out[0].clamp(0.0, 1.0),) + tuple(out[1:])
    return out

Functions:

Gaussian-splat parameter containers (single scene and per-env batch).

Vendored from https://github.com/discoverse-dev/GaussianRenderer.

Classes

GaussianData dataclass 

GaussianData(xyz, rot, scale, opacity, sh)

A single set of Gaussians (shared across all environments).

Source code in eden/utils/batch_splat_renderer/gaussiandata.py 
16
17
18
19
20
21
def __init__(self, xyz, rot, scale, opacity, sh):
    self.xyz = xyz  # (N, 3)
    self.rot = rot  # (N, 4) wxyz
    self.scale = scale  # (N, 3)
    self.opacity = opacity  # (N,)
    self.sh = sh  # (N, K, 3) or (N, 3*K)

Attributes

xyz instance-attribute 
xyz = xyz
rot instance-attribute 
rot = rot
scale instance-attribute 
scale = scale
opacity instance-attribute 
opacity = opacity
sh instance-attribute 
sh = sh
device property 
device

Methods:

to_cuda 
to_cuda()
Source code in eden/utils/batch_splat_renderer/gaussiandata.py 
30
31
32
33
34
35
36
37
def to_cuda(self):
    if not torch.is_tensor(self.xyz) or not self.xyz.is_cuda:
        self.xyz = torch.tensor(self.xyz).float().cuda().requires_grad_(False)
        self.rot = torch.tensor(self.rot).float().cuda().requires_grad_(False)
        self.scale = torch.tensor(self.scale).float().cuda().requires_grad_(False)
        self.opacity = torch.tensor(self.opacity).float().cuda().requires_grad_(False)
        self.sh = torch.tensor(self.sh).float().cuda().requires_grad_(False)
    return self

GaussianBatchData dataclass 

GaussianBatchData(xyz: Tensor, rot: Tensor, scale: Tensor, opacity: Tensor, sh: Tensor)

Per-environment Gaussians produced by :func:batch_update_gaussians.

Attributes

xyz instance-attribute 
xyz: Tensor
rot instance-attribute 
rot: Tensor
scale instance-attribute 
scale: Tensor
opacity instance-attribute 
opacity: Tensor
sh instance-attribute 
sh: Tensor
batch_size property 
batch_size
device property 
device

Methods:

Batched gsplat rasterization: single-scene multi-camera (batch_render) and per-env (batch_env_render).

Vendored from https://github.com/discoverse-dev/GaussianRenderer with Eden edits (tagged # EDEN): torch.as_tensor camera inputs for zero-copy torch-cuda tensors, an optional return_alpha for depth/alpha compositing, and the shared genesis/eden geom helpers in place of the vendored quaternion/point math.

Classes

Functions:

batch_render 

batch_render(gaussians: GaussianData, cam_pos: Tensor, cam_xmat: Tensor, height: int, width: int, fovy: ndarray, bg_imgs: Optional[Tensor] = None, y_up: bool = True, return_alpha: bool = False) -> Tuple[Tensor, ...]

Render one shared Gaussian set from Ncam cameras (shared-scene mode).

Source code in eden/utils/batch_splat_renderer/batch_rasterization.py 
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
@torch.no_grad()
def batch_render(
    gaussians: GaussianData,
    cam_pos: Tensor,  # (Ncam, 3)
    cam_xmat: Tensor,  # (Ncam, 9)
    height: int,
    width: int,
    fovy: np.ndarray,  # (Ncam,) degree
    bg_imgs: Optional[Tensor] = None,  # (Ncam, H, W, 3)
    y_up: bool = True,
    return_alpha: bool = False,  # EDEN
) -> Tuple[Tensor, ...]:
    """Render one shared Gaussian set from ``Ncam`` cameras (shared-scene mode)."""
    device = gaussians.device

    if gaussians.sh.dim() == 2:
        gaussians.sh = gaussians.sh.reshape(gaussians.sh.shape[0], -1, 3).contiguous()
    sh_degree = int(np.round(np.sqrt(gaussians.sh.shape[1]))) - 1

    Ncam = cam_pos.shape[0]
    # EDEN: as_tensor keeps torch-cuda inputs zero-copy; still converts np.
    cam_pos_t = torch.as_tensor(cam_pos, dtype=torch.float32, device=device)  # (N, 3)
    cam_xmat_t = torch.as_tensor(cam_xmat, dtype=torch.float32, device=device).reshape(Ncam, 3, 3)  # (N, 3, 3)
    fovy_t = torch.as_tensor(np.radians(fovy), dtype=torch.float32, device=device)  # (N,)

    # Intrinsics: focal from vertical FOV; square pixels (fx = fy); principal point centered.
    tan_half_fovy = torch.tan(fovy_t / 2.0)
    focal_y = height / (2.0 * tan_half_fovy)
    focal_x = focal_y
    cx = width / 2.0
    cy = height / 2.0

    Ks = torch.zeros((Ncam, 3, 3), dtype=torch.float32, device=device)
    Ks[:, 0, 0] = focal_x
    Ks[:, 1, 1] = focal_y
    Ks[:, 0, 2] = cx
    Ks[:, 1, 2] = cy
    Ks[:, 2, 2] = 1.0

    Tmats = torch.eye(4, dtype=torch.float32, device=device).unsqueeze(0).repeat(Ncam, 1, 1)
    Tmats[:, :3, :3] = cam_xmat_t
    Tmats[:, :3, 3] = cam_pos_t

    # Flip Y and Z columns of rotation (world/OpenGL camera -> gsplat/OpenCV).
    if y_up:
        Tmats[:, 0:3, 1] *= -1
        Tmats[:, 0:3, 2] *= -1
    viewmats = torch.inverse(Tmats)

    renders, alphas, meta = rasterization(
        means=gaussians.xyz,  # [G, 3]
        quats=gaussians.rot,  # [G, 4]
        scales=gaussians.scale,  # [G, 3]
        opacities=gaussians.opacity,  # [G]
        colors=gaussians.sh,  # [G, K, 3]
        viewmats=viewmats,  # [Ncam, 4, 4]
        Ks=Ks,  # [Ncam, 3, 3]
        width=width,
        height=height,
        sh_degree=sh_degree,
        render_mode="RGB+D",
        packed=False,
    )
    color_img = renders[..., :3]  # renders: (Ncam, H, W, 4) RGBD
    depth_img = renders[..., 3:4]

    if bg_imgs is not None:
        if bg_imgs.device != device:
            bg_imgs = bg_imgs.to(device)
        color_img.addcmul_(bg_imgs, 1.0 - alphas)

    if return_alpha:  # EDEN
        return color_img, depth_img, alphas
    return color_img, depth_img

batch_env_render 

batch_env_render(gaussians: GaussianBatchData, cam_pos: Tensor, cam_xmat: Tensor, height: int, width: int, fovy: ndarray, bg_imgs: Optional[Tensor] = None, minibatch: Optional[int] = None, y_up: bool = True, return_alpha: bool = False) -> Tuple[Tensor, ...]

Render per-environment Gaussians (per-env mode).

Each environment carries its own transformed Gaussian set, so the leading Nenv axis is a genuine batch dimension for gsplat (not a camera axis).

Source code in eden/utils/batch_splat_renderer/batch_rasterization.py 
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
@torch.no_grad()
def batch_env_render(
    gaussians: GaussianBatchData,
    cam_pos: Tensor,  # (Nenv, Ncam, 3)
    cam_xmat: Tensor,  # (Nenv, Ncam, 9)
    height: int,
    width: int,
    fovy: np.ndarray,  # (Nenv, Ncam) degree
    bg_imgs: Optional[Tensor] = None,  # (Nenv, Ncam, H, W, 3)
    minibatch: Optional[int] = None,
    y_up: bool = True,
    return_alpha: bool = False,  # EDEN
) -> Tuple[Tensor, ...]:
    """Render per-environment Gaussians (per-env mode).

    Each environment carries its own transformed Gaussian set, so the leading
    ``Nenv`` axis is a genuine batch dimension for gsplat (not a camera axis).
    """
    device = gaussians.device
    Nenv = cam_pos.shape[0]
    Ncam = cam_pos.shape[1]

    if minibatch is not None and minibatch > 0 and minibatch < Nenv:
        out_color = torch.empty((Nenv, Ncam, height, width, 3), dtype=torch.float32, device=device)
        out_depth = torch.empty((Nenv, Ncam, height, width, 1), dtype=torch.float32, device=device)
        out_alpha = (
            torch.empty((Nenv, Ncam, height, width, 1), dtype=torch.float32, device=device) if return_alpha else None
        )

        for i in range(0, Nenv, minibatch):
            end = min(i + minibatch, Nenv)

            g_slice = GaussianBatchData(
                xyz=gaussians.xyz[i:end],
                rot=gaussians.rot[i:end],
                scale=gaussians.scale[i:end],
                opacity=gaussians.opacity[i:end],
                sh=gaussians.sh[i:end],
            )

            bg_slice = bg_imgs[i:end] if bg_imgs is not None else None

            out = batch_env_render(
                g_slice,
                cam_pos[i:end],
                cam_xmat[i:end],
                height,
                width,
                fovy[i:end] if len(fovy) == Nenv else fovy,
                bg_imgs=bg_slice,
                minibatch=None,
                y_up=y_up,
                return_alpha=return_alpha,
            )
            out_color[i:end] = out[0]
            out_depth[i:end] = out[1]
            if return_alpha:
                out_alpha[i:end] = out[2]
        if return_alpha:
            return out_color, out_depth, out_alpha
        return out_color, out_depth

    if gaussians.sh.dim() == 3:  # (Nenv, N, D)
        gaussians.sh = gaussians.sh.reshape(gaussians.sh.shape[0], gaussians.sh.shape[1], -1, 3).contiguous()
    sh_degree = int(np.round(np.sqrt(gaussians.sh.shape[2]))) - 1

    fovy_t = torch.as_tensor(np.radians(fovy), dtype=torch.float32, device=device)  # (Nenv, Ncam)

    # Intrinsics: focal from vertical FOV; square pixels (fx = fy); principal point centered.
    tan_half_fovy = torch.tan(fovy_t / 2.0)
    focal_y = height / (2.0 * tan_half_fovy)
    focal_x = focal_y
    cx = width / 2.0
    cy = height / 2.0

    Ks = torch.zeros((Nenv, Ncam, 3, 3), dtype=torch.float32, device=device)
    Ks[..., 0, 0] = focal_x
    Ks[..., 1, 1] = focal_y
    Ks[..., 0, 2] = cx
    Ks[..., 1, 2] = cy
    Ks[..., 2, 2] = 1.0

    Tmats = torch.eye(4, dtype=torch.float32, device=device).unsqueeze(0).unsqueeze(0).repeat(Nenv, Ncam, 1, 1)
    Tmats[..., :3, :3] = torch.as_tensor(cam_xmat, dtype=torch.float32, device=device).reshape(Nenv, Ncam, 3, 3)
    Tmats[..., :3, 3] = torch.as_tensor(cam_pos, dtype=torch.float32, device=device)

    # Flip Y and Z columns of rotation (world/OpenGL camera -> gsplat/OpenCV).
    if y_up:
        Tmats[..., 0:3, 1] *= -1
        Tmats[..., 0:3, 2] *= -1
    viewmats = torch.inverse(Tmats)

    renders, alphas, meta = rasterization(
        means=gaussians.xyz,  # [Nenv, G, 3]
        quats=gaussians.rot,  # [Nenv, G, 4]
        scales=gaussians.scale,  # [Nenv, G, 3]
        opacities=gaussians.opacity,  # [Nenv, G]
        colors=gaussians.sh,  # [Nenv, G, K, 3]
        viewmats=viewmats,  # [Nenv, Ncam, 4, 4]
        Ks=Ks,  # [Nenv, Ncam, 3, 3]
        width=width,
        height=height,
        sh_degree=sh_degree,
        render_mode="RGB+D",
        packed=False,
    )
    color_img = renders[..., :3]  # renders: (Nenv, Ncam, H, W, 4) RGBD
    depth_img = renders[..., 3:4]

    if bg_imgs is not None:
        if bg_imgs.device != device:
            bg_imgs = bg_imgs.to(device)
        color_img.addcmul_(bg_imgs, 1.0 - alphas)

    if return_alpha:  # EDEN
        return color_img, depth_img, alphas
    return color_img, depth_img

batch_update_gaussians 

batch_update_gaussians(gaussian_template: GaussianData, body_pos: Tensor, body_quat: Tensor, point_to_body_idx: Optional[Tensor], dynamic_mask: Optional[Tensor], scalar_first: bool = True) -> GaussianBatchData

Rigidly transform template Gaussians by each env's per-body poses.

xyz/rot are materialized per env: static points (dynamic_mask == False, e.g. background) keep the template's, dynamic points are moved by their body's pose. Scale, opacity and SH are broadcast across envs as strided views.

Source code in eden/utils/batch_splat_renderer/batch_rasterization.py 
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
@torch.no_grad()
def batch_update_gaussians(
    gaussian_template: GaussianData,
    body_pos: Tensor,  # (Nenv, Nbody, 3)
    body_quat: Tensor,  # (Nenv, Nbody, 4)
    point_to_body_idx: Optional[Tensor],  # (N_points,)
    dynamic_mask: Optional[Tensor],  # (N_points,)
    scalar_first: bool = True,
) -> GaussianBatchData:
    """Rigidly transform template Gaussians by each env's per-body poses.

    xyz/rot are materialized per env: static points (``dynamic_mask == False``,
    e.g. background) keep the template's, dynamic points are moved by their
    body's pose. Scale, opacity and SH are broadcast across envs as strided views.
    """
    device = body_pos.device
    Nenv = body_pos.shape[0]
    Ngs = len(gaussian_template)

    # Materialize a numpy template to tensors once (cache in practice).
    if isinstance(gaussian_template.xyz, np.ndarray):
        tmpl_xyz = torch.tensor(gaussian_template.xyz, dtype=torch.float32, device=device)
        tmpl_rot = torch.tensor(gaussian_template.rot, dtype=torch.float32, device=device)
        tmpl_scale = torch.tensor(gaussian_template.scale, dtype=torch.float32, device=device)
        tmpl_opacity = torch.tensor(gaussian_template.opacity, dtype=torch.float32, device=device)
        tmpl_sh = torch.tensor(gaussian_template.sh, dtype=torch.float32, device=device)
    else:
        tmpl_xyz = gaussian_template.xyz
        tmpl_rot = gaussian_template.rot
        tmpl_scale = gaussian_template.scale
        tmpl_opacity = gaussian_template.opacity
        tmpl_sh = gaussian_template.sh

    xyz_out = tmpl_xyz.unsqueeze(0).expand(Nenv, Ngs, 3).clone()
    rot_out = tmpl_rot.unsqueeze(0).expand(Nenv, Ngs, 4).clone()

    mask = dynamic_mask
    if mask is not None and mask.any():
        body_indices = point_to_body_idx[mask]  # (N_dynamic,)

        # Gather body poses per env: (Nenv, N_dynamic, 3/4)
        pos_expanded = body_pos[:, body_indices]
        quat_expanded = body_quat[:, body_indices]

        # Template properties: (N_dynamic, 3/4) -> (1, N_dynamic, 3/4)
        xyz_ori = tmpl_xyz[mask].unsqueeze(0)
        rot_ori = tmpl_rot[mask].unsqueeze(0)

        if not scalar_first:
            quat_expanded = quat_expanded[..., [3, 0, 1, 2]]  # xyzw -> wxyz

        # Apply transform (broadcasts (1, N_dyn, .) with (Nenv, N_dyn, .))
        xyz_new = transform_by_quat(xyz_ori, quat_expanded) + pos_expanded  # EDEN: was transform_points
        rot_new = quat_mul(quat_expanded, rot_ori)  # EDEN: was quaternion_multiply

        xyz_out[:, mask, :] = xyz_new
        rot_out[:, mask, :] = rot_new

    # Static properties as strided views (not materialized).
    scale_out = tmpl_scale.unsqueeze(0).expand(Nenv, Ngs, 3)
    opacity_out = tmpl_opacity.unsqueeze(0).expand(Nenv, Ngs)
    sh_out = tmpl_sh.unsqueeze(0).expand(Nenv, Ngs, -1, 3)

    return GaussianBatchData(xyz=xyz_out, rot=rot_out, scale=scale_out, opacity=opacity_out, sh=sh_out)

PLY files

Load standard 3DGS / 2DGS .ply files into a :class:GaussianData.

Adapted from https://github.com/discoverse-dev/GaussianRenderer (load_ply_3dgs), trimmed to the standard binary 3DGS/2DGS path (the upstream SuperSplat loader is not vendored).

Classes

Functions:

load_ply 

load_ply(path: str, *, activate: bool = True) -> GaussianData

Load a 3DGS/2DGS PLY into a numpy :class:GaussianData (sh as (N, K, 3)).

PARAMETER DESCRIPTION
path

Path to a standard binary 3DGS/2DGS .ply (x,y,z, f_dc_*, f_rest_*, opacity, scale_*, rot_*).

TYPE: str

activate

Apply the stored-parameter activations (exp on scale, sigmoid on opacity, quaternion normalization) so the returned values are the ones gsplat consumes directly. Set False to inspect raw stored params.

TYPE: bool DEFAULT: True

RETURNS DESCRIPTION
GaussianData

xyz (N,3), rot (N,4) wxyz, scale (N,3), opacity (N,), sh (N, K, 3) with K = (sh_degree + 1) ** 2. A 2DGS PLY (only two scale_* fields) gets a near-zero third scale so it reads as a flat 3DGS ellipsoid.
Source code in eden/utils/batch_splat_renderer/ply_io.py 
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
def load_ply(path: str, *, activate: bool = True) -> GaussianData:
    """Load a 3DGS/2DGS PLY into a numpy :class:`GaussianData` (sh as ``(N, K, 3)``).

    Parameters
    ----------
    path : str
        Path to a standard binary 3DGS/2DGS ``.ply`` (``x,y,z``, ``f_dc_*``,
        ``f_rest_*``, ``opacity``, ``scale_*``, ``rot_*``).
    activate : bool
        Apply the stored-parameter activations (``exp`` on scale, ``sigmoid`` on
        opacity, quaternion normalization) so the returned values are the ones
        gsplat consumes directly. Set ``False`` to inspect raw stored params.

    Returns
    -------
    GaussianData
        ``xyz (N,3)``, ``rot (N,4) wxyz``, ``scale (N,3)``, ``opacity (N,)``,
        ``sh (N, K, 3)`` with ``K = (sh_degree + 1) ** 2``. A 2DGS PLY (only two
        ``scale_*`` fields) gets a near-zero third scale so it reads as a flat
        3DGS ellipsoid.
    """
    plydata = PlyData.read(path)
    el = plydata.elements[0]
    props = el.properties

    xyz = np.stack([np.asarray(el["x"]), np.asarray(el["y"]), np.asarray(el["z"])], axis=1).astype(np.float32)
    opacities = np.asarray(el["opacity"]).astype(np.float32)

    features_dc = np.stack(
        [np.asarray(el["f_dc_0"]), np.asarray(el["f_dc_1"]), np.asarray(el["f_dc_2"])], axis=1
    )  # (N, 3)

    extra_names = _sorted_props(props, "f_rest_")
    n_extra = len(extra_names)
    if n_extra % 3 != 0:
        raise ValueError(f"{path}: expected f_rest_* count divisible by 3, got {n_extra}")
    n_coeffs = n_extra // 3  # rest coefficients per channel
    sh_degree = int(round(np.sqrt(n_coeffs + 1))) - 1
    if (sh_degree + 1) ** 2 != n_coeffs + 1:
        raise ValueError(f"{path}: f_rest count {n_extra} is not a valid SH degree (deg1=9, deg2=24, deg3=45)")

    features_extra = np.zeros((xyz.shape[0], n_extra), dtype=np.float32)
    for idx, name in enumerate(extra_names):
        features_extra[:, idx] = np.asarray(el[name])
    # (N, 3, n_coeffs) -> (N, n_coeffs, 3): channel-major storage -> coeff-major
    features_extra = features_extra.reshape(xyz.shape[0], 3, n_coeffs).transpose(0, 2, 1)

    scale_names = _sorted_props(props, "scale_")
    scales = np.stack([np.asarray(el[n]) for n in scale_names], axis=1).astype(np.float32)

    rot_names = _sorted_props(props, "rot")
    rots = np.stack([np.asarray(el[n]) for n in rot_names], axis=1).astype(np.float32)

    if activate:
        rots = rots / np.linalg.norm(rots, axis=-1, keepdims=True)
        scales = np.exp(scales)
        opacities = 1.0 / (1.0 + np.exp(-opacities))

    # 2DGS surfel: only two in-plane scales. Pad a near-flat third axis so the
    # 3DGS rasterizer treats it as a thin disk (see this package's README: 2DGS assets).
    if scales.shape[1] == 2:
        scales = np.hstack([scales, np.full((scales.shape[0], 1), 1e-9, dtype=np.float32)])

    # sh: (N, K, 3) with the DC coeff first, then the rest coeffs.
    sh = np.concatenate([features_dc[:, None, :], features_extra], axis=1).astype(np.float32)  # (N, K, 3)

    return GaussianData(
        xyz=xyz,
        rot=rots.astype(np.float32),
        scale=scales.astype(np.float32),
        opacity=opacities.astype(np.float32),
        sh=sh,
    )

Utility to convert gaussian splat PLY files to a renderer-compatible format.

The Nyx light-field importer requires SH degree 3 PLY files with f_rest_0..f_rest_44. Some PLY files (e.g. from WorldLabs) only carry SH degree 0 (f_dc_0..f_dc_2, no f_rest_*). This utility pads the missing SH coefficients with zeros so the resulting PLY is accepted by the renderer.

Functions:

needs_sh_padding 

needs_sh_padding(file_path: str) -> bool

Check if a PLY file needs SH coefficient padding for renderer compatibility.

Source code in eden/utils/splat_ply_converter.py 
37
38
39
40
41
42
43
def needs_sh_padding(file_path: str) -> bool:
    """Check if a PLY file needs SH coefficient padding for renderer compatibility."""
    _, properties, _, _ = _parse_ply_header(file_path)
    prop_names = {name for _, name in properties}
    has_f_rest = any(name.startswith("f_rest_") for name in prop_names)
    has_f_dc = "f_dc_0" in prop_names
    return has_f_dc and not has_f_rest

convert_to_sh3_ply 

convert_to_sh3_ply(input_path: str, output_path: str | None = None, sh_degree: int = 3) -> str

Convert a gaussian splat PLY to SH-degree-3 format by padding SH coefficients.

The input must be a binary little-endian PLY (the output header always declares that format).

PARAMETER DESCRIPTION
input_path

Path to the input PLY file.

TYPE: str

output_path

Path to write the output PLY. If None, writes to a temp file.

TYPE: str DEFAULT: None

sh_degree

Target SH degree. Default 3 (requires f_rest_0..f_rest_44).

TYPE: int DEFAULT: 3

RETURNS DESCRIPTION
str

Path to the output PLY file (same as input if no conversion needed).
Source code in eden/utils/splat_ply_converter.py 
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
def convert_to_sh3_ply(input_path: str, output_path: str | None = None, sh_degree: int = 3) -> str:
    """Convert a gaussian splat PLY to SH-degree-3 format by padding SH coefficients.

    The input must be a binary little-endian PLY (the output header always declares that format).

    Parameters
    ----------
    input_path : str
        Path to the input PLY file.
    output_path : str, optional
        Path to write the output PLY. If None, writes to a temp file.
    sh_degree : int
        Target SH degree. Default 3 (requires f_rest_0..f_rest_44).

    Returns
    -------
    str
        Path to the output PLY file (same as input if no conversion needed).
    """
    if not needs_sh_padding(input_path):
        return input_path

    _, properties, vertex_count, header_end_offset = _parse_ply_header(input_path)

    num_sh_rest = 3 * (sh_degree + 1) ** 2 - 3  # 45 for degree 3

    new_properties = []
    for dtype, name in properties:
        new_properties.append((dtype, name))
        if name == "f_dc_2":
            for i in range(num_sh_rest):
                new_properties.append(("float", f"f_rest_{i}"))

    type_sizes = {"float": 4, "double": 8, "uchar": 1, "int": 4, "uint": 4, "short": 2, "ushort": 2}
    old_vertex_size = sum(type_sizes.get(dtype, 4) for dtype, _ in properties)
    padding_size = num_sh_rest * 4

    insert_byte_offset = 0
    for dtype, name in properties:
        insert_byte_offset += type_sizes.get(dtype, 4)
        if name == "f_dc_2":
            break

    new_header_lines = []
    new_header_lines.append("ply")
    new_header_lines.append("format binary_little_endian 1.0")
    new_header_lines.append(f"element vertex {vertex_count}")
    for dtype, name in new_properties:
        new_header_lines.append(f"property {dtype} {name}")
    new_header_lines.append("end_header")
    new_header = "\n".join(new_header_lines) + "\n"

    if output_path is None:
        fd, output_path = tempfile.mkstemp(suffix=".ply", prefix="splat_sh3_")
        os.close(fd)

    zero_padding = b"\x00" * padding_size

    with open(input_path, "rb") as fin, open(output_path, "wb") as fout:
        fin.seek(header_end_offset)
        fout.write(new_header.encode("ascii"))
        for _ in range(vertex_count):
            vertex_data = fin.read(old_vertex_size)
            fout.write(vertex_data[:insert_byte_offset])
            fout.write(zero_padding)
            fout.write(vertex_data[insert_byte_offset:])

    return output_path

Background shadows

Carry the shadows the sim casts on background surfaces over to a composited background.

A camera's background composite (:class:~eden.entities.camera_overlays.BackgroundComposite) replaces the pixels of its entities with another image (a static background image or a Gaussian-splat render). The replacement is by segmentation mask, so the shadows the robot and objects cast onto those surfaces in the sim pass are thrown away with them, and the objects look like they float above the real table.

:func:background_shadow_factor recovers those shadows from the sim pass itself. Each background surface is still rendered (with shadows) before it is replaced, so its brightness divided by its brightness without shadows is the fraction of light that reached it. The result is a per-pixel multiplier for the replacement image. The brightness without shadows comes from one of two places:

  • a second render of the same view with shadows turned off (unshadowed_rgb). This is exact for any surface, texture and lighting, and is what the rasterizer camera uses;
  • otherwise, an estimate: a high quantile of each surface's brightness in view, per segmentation id and per env. This is exact for a flat, untextured surface with diffuse shading under directional lights (every lit pixel is then equally bright), which is what the batch renderer draws; a brightness falloff across the surface (a point light, a highlight, a texture) is read as shadow. It also needs part of each surface to be lit in every view: the reference is taken from the brightest 1 - _LIT_QUANTILE (3 %) of the surface's pixels, so when more of the surface than that is in shadow (a wrist camera looking at the table under the gripper) a shadowed pixel becomes the reference and the shadow is lost. The camera rejects a textured surface at build time, since that one is known up front; the lit fraction depends on the view and cannot be.

Functions:

background_shadow_factor 

background_shadow_factor(rgb: Tensor, segm: Tensor, surface_ids: Sequence[int] | Tensor, strength: float, unshadowed_rgb: Tensor | None = None) -> Tensor

Per-pixel multiplier that puts the sim's shadows on a replacement background.

PARAMETER DESCRIPTION
rgb

Sim-pass colour, (H, W, C) or (N, H, W, C) with C >= 3, any numeric dtype.

TYPE: Tensor

segm

Segmentation ids matching rgb, (H, W) or (N, H, W).

TYPE: Tensor

surface_ids

Segmentation ids of the background surfaces that receive shadows. Each id is normalised on its own, so surfaces of different colours can share a frame. A tensor on segm's device is used without a host-to-device copy (a caller rendering every frame should keep one rather than pass a list); it is only cast if its dtype differs from segm's.

TYPE: Sequence[int] | Tensor

strength

How much a fully shadowed pixel is darkened, in [0, 1]: 0 leaves the background unchanged, 1 turns a pixel that receives no light black.

TYPE: float

unshadowed_rgb

The same view rendered without shadows, shaped like rgb. When given, each pixel is compared with its own unshadowed brightness; when None, each surface's unshadowed brightness is estimated from rgb (see the module docstring).

TYPE: Tensor | None DEFAULT: None

RETURNS DESCRIPTION
Tensor

Float multiplier in [1 - strength, 1] with shape segm.shape + (1,), equal to 1 outside the surfaces.
Source code in eden/utils/background_shadows.py 
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
def background_shadow_factor(
    rgb: torch.Tensor,
    segm: torch.Tensor,
    surface_ids: Sequence[int] | torch.Tensor,
    strength: float,
    unshadowed_rgb: torch.Tensor | None = None,
) -> torch.Tensor:
    """Per-pixel multiplier that puts the sim's shadows on a replacement background.

    Parameters
    ----------
    rgb : torch.Tensor
        Sim-pass colour, ``(H, W, C)`` or ``(N, H, W, C)`` with ``C >= 3``, any numeric dtype.
    segm : torch.Tensor
        Segmentation ids matching ``rgb``, ``(H, W)`` or ``(N, H, W)``.
    surface_ids : Sequence[int] | torch.Tensor
        Segmentation ids of the background surfaces that receive shadows. Each id is normalised on
        its own, so surfaces of different colours can share a frame. A tensor on ``segm``'s device is
        used without a host-to-device copy (a caller rendering every frame should keep one rather than
        pass a list); it is only cast if its dtype differs from ``segm``'s.
    strength : float
        How much a fully shadowed pixel is darkened, in ``[0, 1]``: ``0`` leaves the background
        unchanged, ``1`` turns a pixel that receives no light black.
    unshadowed_rgb : torch.Tensor | None
        The same view rendered without shadows, shaped like ``rgb``. When given, each pixel is compared
        with its own unshadowed brightness; when ``None``, each surface's unshadowed brightness is
        estimated from ``rgb`` (see the module docstring).

    Returns
    -------
    torch.Tensor
        Float multiplier in ``[1 - strength, 1]`` with shape ``segm.shape + (1,)``, equal to ``1``
        outside the surfaces.
    """
    if not 0.0 <= strength <= 1.0:
        raise ValueError(f"strength must be in [0, 1], got {strength}.")
    if rgb.shape[:-1] != segm.shape:
        raise ValueError(f"rgb {tuple(rgb.shape)} and segm {tuple(segm.shape)} do not cover the same pixels.")
    if unshadowed_rgb is not None and unshadowed_rgb.shape != rgb.shape:
        raise ValueError(f"unshadowed_rgb {tuple(unshadowed_rgb.shape)} does not match rgb {tuple(rgb.shape)}.")

    n_envs = 1 if segm.ndim == 2 else segm.shape[0]
    # Channel sums accumulate in float32 directly: `.float()` first would materialise a float copy
    # of the whole (N, H, W, C) frame.
    brightness = rgb[..., :3].sum(dim=-1, dtype=torch.float32).reshape(n_envs, -1)
    segm = segm.reshape(n_envs, -1)
    surface_ids = torch.as_tensor(surface_ids, device=segm.device, dtype=segm.dtype)
    on_surface = torch.isin(segm, surface_ids)

    if unshadowed_rgb is not None:
        lit = unshadowed_rgb[..., :3].sum(dim=-1, dtype=torch.float32).reshape(n_envs, -1)
    else:
        lit = torch.zeros_like(brightness)
        for surface_id in surface_ids:
            mask = segm == surface_id
            lit = torch.where(mask, _masked_quantile(brightness, mask, _LIT_QUANTILE)[:, None], lit)
    # A pixel that is black even without shadows carries no information about light: leave it alone.
    keep = ~(on_surface & (lit > 0))
    # Both are channel sums, so their ratio is the ratio of the channel means.
    light = brightness.div_(lit.clamp_(min=1e-6)).clamp_(max=1.0).masked_fill_(keep, 1.0)

    # 1 - strength * (1 - light), in place.
    return light.mul_(strength).add_(1.0 - strength).reshape(*rgb.shape[:-1], 1)

See also