A small number of USD files in the wild contain invalid face index data
for some of their meshes. This leads to asserts in debug builds and
crashes for users in retail builds(sometimes). There is already an
import option to Validate Meshes but it turns out that we, and most
other importers, perform validation too late. We crash before getting to
that validate option (see notes).
This PR implements a cheap detection mechanism and will auto-fix if we
detect broken data. The detection may not find all types of bad data but
it will detect what is known to fail today for duplicate vertex indices.
We immediately validate/fix before loading in the rest of the data. The
downside is that this will mean no additional data will be loaded.
Normals, edge creases, velocities, UVs, and all other attributes will be
lost because the incoming data arrays will no longer align.
It should be noted also that Alembic has also chosen this approach. It's
check is significantly weaker though and can be improved separately if
needed.
If auto-fix is triggered, it will typically appear as one trace on the
terminal.
```
WARN (io.usd): <path...>\io\usd\intern\usd_reader_mesh.cc:684
read_mesh_sample: Invalid face data detected for mesh
'/degenerate/m_degenerate'. Automatic correction will be used.
```
A more general downside of these fixes is that this applies to each
frame of animated mesh data. The mesh will be fixed, and re-fixed, on
every frame update when the frame in question contains bad data.
For well-behaved USD scenes, the penalty for this check is between 2-4%.
For broken USD scenes, it depends on how many meshes need the fixup. In
the case of the Intel 4004 Moore Lane scene, the penalty is a 2.7x
slowdown in import time (4.5 s to 12.5 s).
Pull Request: https://projects.blender.org/blender/blender/pulls/138633
Handle the `DomeLight_1` schema for import and translate to a World
material like what was already done for the original `DomeLight` schema.
The primary difference is that the new schema provides a `poleAxis`
attribute that authoring applications can use to remove ambiguity for
the orientation of the HDRI texture. Some care was made to match
`usdview` with a set of hand-crafted files. However, after matching,
some real scenes ended up displaying differently. These were corrected
but this could mean there's still issues that will need investigation
and fixing in the future.
Co-authored-by: Nig3l <nig3lpro@gmail.com>
Co-authored-by: Jesse Yurkovich <jesse.y@gmail.com>
Pull Request: https://projects.blender.org/blender/blender/pulls/137761
Add support for the UsdPrimvarReader_TYPE templates for both import and
export. These are used by several USD test assets and support here
represents the last major piece of the UsdPreviewSurface spec to be
implemented.
On import these become `Attribute` nodes and on export the `Attribute`
nodes will become `UsdPrimvarReader_TYPE`'s accordingly.
Import:
- `UsdPrimvarReader_float` and `UsdPrimvarReader_int` will use the `Fac`
output
- `UsdPrimvarReader_float3` and `UsdPrimvarReader_float4` will use the
`Color` output
- `UsdPrimvarReader_vector`, `UsdPrimvarReader_normal`, and
`UsdPrimvarReader_point` will use the `Vector` output
Export (only `Geometry` Attribute types are considered):
- `Fac` will use `UsdPrimvarReader_float`
- `Color` will use `UsdPrimvarReader_float3`
- `Vector` will use `UsdPrimvarReader_vector`
- `Alpha` is not considered
MaterialX note:
Hydra-native support is a bit more involved and will have to be done
separately. Hydra w/USD sync is trivial to implement but those changes
have been left out here.
Pull Request: https://projects.blender.org/blender/blender/pulls/135143
Imports in the following animated data from UsdGeomCameras:
- Focal length
- DOF distance
- DOF fstop
- Clip start, Clip end
- Tilt shift x, Tilt shift y
- Aperture sizes (with caveats)
Implementation wise, it's more complicated than I'd like due to needing
to read in both non-animated and animated data for each property. And
because I've tried to reduce the duplication of various transforms we
have to do on each value. E.g. scaling values by "tenth of scene units"
or the extraction of the USD clipping range from a GfVec2f into 2
separate properties, and 2 separate fcurves, in Blender etc. The current
approach was the best I could come up with so far.
Aperture sizes remain problematic for import, with animation data and
without, due to how Blender selects the largest sensor dimension to base
downstream calculations on and for which there's no concept in USD to
strictly dictate which dimension to use. Additionally, changing the
sensor size will impact the Tilt values as well. This means that if the
Aperture sizes are animated, we must also animate the tilt values; leads
to more fcurves being created than perhaps expected.
The `projection` attribute (perspective and orthographic) remains
unchanged (non animated only) due to differences in how USD<>Blender
interoperate with the Orthographic projection method. Note: Blender only
exports perspective cameras due to the same reason.
Pull Request: https://projects.blender.org/blender/blender/pulls/137487
Instead of converting to a string immediately, only convert to a string
when necessary, like for trace logging. This reduces the size of the
`USDPrimReader` base class by 24 bytes, makes several Vectors and Maps
smaller at runtime, and reduces some unnecessary string allocations and
conversions in various other places.
The recently added python Hook `get_prim_path` dictionary will now
contain `SdfPath` objects rather than strings; removing the string
conversion cost when calling other USD APIs like `GetPrimAtPath` which
take in `SdfPath` objects to begin with.
This also makes the code a bit more self explanatory since the type
makes it clear what kind of path you are dealing with.
Pull Request: https://projects.blender.org/blender/blender/pulls/133954
This pairs with the recently added Export test and reuses the same
.blend file. While there happens to be an instancing related rendering
test, that happens to cover Import, it's better to have a direct test
for it.
Future improvements to both Export and Import to account for the recent
"hidden layer" visibility bug would be good to do as a follow-up.
Pull Request: https://projects.blender.org/blender/blender/pulls/133782
Makes use of recently added test data to ensure proper handling of child
objects, with animated transform constraints, parented to other objects
who also have animated transform constraints.
Also uses the `colored_print` module to better segment the test output.
Pull Request: https://projects.blender.org/blender/blender/pulls/133600
Export
Like we do for Mesh and PointCloud, export any "velocity" attribute on
the Point domain as native USD "velocities". While testing, a few
additional blender-internal attributes were discovered being exported
which are now excluded during export.
Import
Add the cache modifier as appropriate when we detect that UsdBasisCurve
data is animated. This includes time-varying positions, widths,
velocities, and general attribute values. Before this PR, only the
positions were considered. And like Export, the native USD "velocities"
attribute is now processed.
Adds test coverage as well.
Pull Request: https://projects.blender.org/blender/blender/pulls/133027
This rescales the whole scene by its root transform to match the same
visual size while not forcing the user to wait for scale to be applied to
each object.
This is requested by studios whose main applications / USD scenes are
in CM, because referencing and payloading scenes from disparate scales
can cause issues at resolution time.
If "Apply Unit Scale Conversion" is unchecked on import, the user now
has the ability to bring the objects in with a scale factor of 1.0, so that the
objects may be edited as if Blender's scene units matches the imported
stage's.
At export time, a "Stage Meters Per Unit" value can be chosen from a list
of common measurements, as well as setting a custom value.
Co-authored-by: kiki <charles@skeletalstudios.com>
Co-authored-by: Michael Kowalski <makowalski@nvidia.com>
Pull Request: https://projects.blender.org/blender/blender/pulls/122804
Importing USD `quatf` types was erroneously left disabled after recent
work this past summer for better attribute support. It is already
correctly enabled and validated for Export.
The `half` and `double` variants must still be skipped for now though.
Pull Request: https://projects.blender.org/blender/blender/pulls/132252
Import edge crease values and properly configure any added subdivision
modifiers with correct UV and boundary settings. These were already
exported so this now completes our subdivision support.
Pull Request: https://projects.blender.org/blender/blender/pulls/131569
Until the newer Hair Curves system can fully replace particle hair, add
a small test to ensure this continues to work.
Since the hair is exported as cubic bspline curves, we can also use this
same file to test bspline import now too.
Pull Request: https://projects.blender.org/blender/blender/pulls/131997
Supporting a new on_material_import() USDHook callback.
Also added support for import_texture() and export_texture()
utility functions which can be called from on_material_import() and
on_material_export() Python implementations, respectively.
Pull Request: https://projects.blender.org/blender/blender/pulls/131559
When importing an USD, the Blender object names do not necessarily match
Prim names:
```
#usda 1.0
def Xform "xform1"
{
def Mesh "MyObject" # Will be imported as "MyObject"
}
def Xform "xform2"
{
def Mesh "MyObject" # Will be imported as "MyObject.001"
}
def Xform "xform2"
{
def Mesh "MyObject" # Will be imported as "MyObject.002"
}
```
This makes it difficult in the [USD Import Hooks]
(https://docs.blender.org/api/current/bpy.types.USDHook.html) to link a
Blender object back to its source Prim for additional processing. A
typical use cases for games is to generate UVs, create and apply a
material on the fly when importing a collision shape that does not have
a visual representation (hence no materials) based on some Prim
attributes, but that the artist needs to differenciate in Blender's
viewport.
The Import context exposes a new method `get_prim_map()` that returns a
`dict` of `prim path` / `list of Blender ID`.
For example, given the following USD scene,
```
/
|--XformThenCube [def Xform]
| `--Cube [def Cube]
|--XformThenXformCube [def Xform]
| `--XformIntermediate [def Xform]
| `--Cube [def Mesh]
|--Cube [def Cube]
`--Material [def Material]
`--Principled_BSDF [def Shader]
```
the `get_prim_map()` method will return a map as:
```python
@static_method
def on_import(import_context):
pprint(import_context.get_prim_map())
```
```json
{
"/Cube": [bpy.data.objects["Cube.002"], bpy.data.meshes["Cube.002"]],
"/XformThenCube": [bpy.data.objects["XformThenCube"]],
"/XformThenCube/Cube": [bpy.data.objects["Cube"], bpy.data.meshes["Cube"]],
"/XformThenXformCube": [bpy.data.objects["XformThenXformCube"]],
"/XformThenXformCube/XformIntermediate": [bpy.data.objects["XformIntermediate"]],
"/XformThenXformCube/XformIntermediate/Cube": [bpy.data.objects["Cube.001"], bpy.data.meshes["Cube.001"]],
"/Material": [bpy.data.materials["Material"]],
}
```
Co-authored-by: Odréanne Breton <odreanne.breton@ubisoft.com>
Co-authored-by: Sttevan Carnali Joga <sttevan.carnali-joga@ubisoft.com>
Co-authored-by: Charles Flèche <charles.fleche@ubisoft.com>
While adding tests for subd import I discovered that our vertex crease
data was imported incorrectly.
This PR adds tests and fixes:
- We tried to read crease sharpness data as ints instead of floats. This
caused the import process to early-return, meaning we never load any
data at all
- Empty USD data would still cause us to create the `crease_vert`
attribute unnecessarily
- Sharpness data needs clamped to 0-1 to handle USD's SHARPNESS_INFINITE
value of 10.0
- We need to fill the `crease_vert` span with 0's since incoming USD
data may not have values set for all the verts
Pull Request: https://projects.blender.org/blender/blender/pulls/131516
Rather than using a default "Anim" name when exporting armature
animations, use the animation's Action name instead. Likewise use the
UsdSkelAnimation name for the Action's name during import.
Blendshapes/shapekeys still use the default "Anim" name as there's no
action for these situations.
Pull Request: https://projects.blender.org/blender/blender/pulls/131021
When EEVEE next removed the Alpha Threshold option a node graph setup
was used instead. Based on a USD test asset[1] the original code was
designed around only using the Image A channel during import to wire
up the nodes. However, the other channels (R, G, or B) are just as
valid.
We now handle this by wiring up the Separate RGB node, that is already
inserted for the scenario in general, to the shader nodes handling the
threshold calculation. Also adds additional test coverage for both
export and import as well.
The Round and LessThan+OneMinus networks remain the only shapes that
will be recognized for the processing.
[1] https://github.com/usd-wg/assets/tree/main/test_assets/AlphaBlendModeTest
Pull Request: https://projects.blender.org/blender/blender/pulls/130346
The existing point instancer reader is slightly refactored to allow for
animated setups. The primary change is simply to inherit from
`USDGeomReader` rather than `USDXformReader`. This allows access to the
`read_geometry` API used by the cache modifier.
The existing `read_object_data` method is split into two parts with
`read_geometry` loading per-frame USD data and `read_object_data`
coordinating the initial loading process, including creating the GN node
tree just once.
A new test has been added (a variation of the nested point instancer
file) with time samples on various attributes and on both point
instancers. This also fixes#129502 and the file provided in that issue.
----
The already added test file is `tests/data/usd/usd_point_instancer_anim.usda`
Pull Request: https://projects.blender.org/blender/blender/pulls/129881
This new import option will allow users to chose if we merge USD prims
with their Xform parent.
Given a USD like:
```
def Xform "MyObject"
{
def Mesh "MyObject_LOD0"
{
}
}
```
When the option is set to True (existing default), only the mesh will be
imported and its parent Xform transformation will be baked into the
Mesh.
```
# In blender after import
- MyObject_LOD0 (Mesh)
```
When the option is set to False, the parent Xform will always be
imported as an Empty object:
```
# In blender after import
- MyObject (Empty)
├─ MyObject_LOD0 (Mesh)
```
Co-authored-by: Odréanne Breton <odreanne.breton@ubisoft.com>
Co-authored-by: Sttevan Carnali Joga <sttevan.carnali-joga@ubisoft.com>
This enables material displacement for UsdPreviewSurface import and
export. Scenarios are limited by what's supported by the preview surface
itself. Namely only Object Space displacement can be used (no vector
displacement)[1] and the Midlevel and Scale parameters are maintained by
adjusting the scale-bias on the image texture controlling the Height
(this means that Midlevel and Scale must be constants).
Hydra/MaterialX support is more complicated. First, there is a bug which
prevents scalar displacment from working correctly and that needs USD
2408+ for the fix[2]. Second, is that there's an open question about
which coordinate system to use for MaterialX's vector displacement maps.
Lastly, Hydra GL does not render displacement, making verification using
only Blender impossible[3]. As a result, this PR only makes MaterialX
"ready" for support, but stops short of actually connecting the final
piece of the node graph until more of the above can be sorted out.
Tests are added which cover:
- Variations of Midlevel and Scale values
- A constant Height setup
- Negative scenarios checking that only Object space is supported
and that midlevel and scale need to be constants
[1] https://openusd.org/release/spec_usdpreviewsurface.html
[2] https://github.com/PixarAnimationStudios/OpenUSD/issues/3325
[3] https://forum.aousd.org/t/materialx-displacement-hydra-storm/1098/2
Pull Request: https://projects.blender.org/blender/blender/pulls/128909
The existing Volume export, which already supports VDB file sequences
and static volumes created inside Blender, is now extended to handle
dynamically created and modified volumes. This allows scenarios where a
Volume Displace modifier is placed over-top an existing VDB sequence or
when Geometry Nodes is used to create animated volumes procedurally.
Detection of what counts as animation is simplistic and mimics what has
been used for Meshes. Essentially if there are any modifiers on the
volume we assume that the volume is "varying" in some way. This can lead
to situations where new volume files are written unnecessarily.
Volume import was also adjusted to correctly set the sequence "offset"
value. This is required to properly handle the case when a VDB sequence
begins animating at a different frame than what's implied by the file
name. For example, a VDB file sequence with file names containing 14-19
but the user wants to animate on frames 8-13 instead.
Tests are added which cover:
- Animated VDB file sequences
- Animated Mesh To Volume where the mesh has been animated
- Animated Volume Displacement where displacement settings are animated
- Animated Volumes created with a Geometry Nodes simulation
----
New test data has been checked in: `tests/data/usd/usd_volumes.blend` and files inside `tests/data/usd/volume-data/`
Pull Request: https://projects.blender.org/blender/blender/pulls/128907
When using the `create_collection` parameter during import, the newly
created Collection would be assigned a fake user which isn't necessary
and caused the user count to be 2 instead of the more natural 1 here.
Remove the fake user and add test coverage for the scenario in general.
Pull Request: https://projects.blender.org/blender/blender/pulls/128348
This includes tests for a "static" mesh where the materials have been
assigned and are expected to remain the same over time.
The test scenario also includes a "dynamic" mesh where both new faces
and new materials are assigned to the same mesh over time (inspired by
#118754). However, that cannot be tested right now due to missing
features of both the MeshSequenceCache and our USD IO code. No testing
will occur for that case until the features are implemented.
Pull Request: https://projects.blender.org/blender/blender/pulls/128341
An additional scenario was added which tests multiple meshes with shape
keys being animated by a single armature. The meshes purposely use the
same shape key name to ensure we also correctly account for that.
Additionally, because the existing shape key and armature test coverage
comes from already written .usd files, this test will first export from
Blender so we can better validate a full Blender-to-Blender roundtrip.
Pull Request: https://projects.blender.org/blender/blender/pulls/128160
Color primvars/attributes were historically treated as a special case
for both import and export. This was mostly done to align with how
painting and viewport display works in Blender. Export would generally
ignore color attributes except when they were found on a Mesh's Point or
FaceCorner domains. And import went out of its way to map incoming color
primvars to the FaceCorner domain in more situations than necessary.
To facilitate better roundtripping in Blender<=>USD workflows, and to
reduce code duplication, this PR teaches the common attribute utilities
how to handle color types. The color attributes will now work on all
relevant Mesh and Curve domains.
There were tests in place for this already but they were set to verify
the inverse state, i.e. the technically broken state, until this could
be fixed.
There remains one special case: "displayColor" primvars and attributes.
The "displayColor" is a special primvar in USD and is the de-facto way
to set a simple viewport color in that ecosystem. It must also be a
color3f type. In order to not regress import, if a "displayColor"
primvar is found on the Face domain we will map it to FaceCorner instead
so it can be displayed in the viewport; which has been the case for the
past several releases. We can drop this special-case if/when Blender can
display Face colors through the Viewport Shading "Attribute" color type.
Additionally, Blender will export this, and only this, color attribute
as a color3f.
Note: As was the case prior to this PR, the following 2 discrepancies
still prevent "perfect" round-trips:
- USD does not have an equivalent to Blender's byte colors; they are
treated as float during IO
- Blender does not have an equivalent to USD's color3 types; they are
treated as color4 during IO
Pull Request: https://projects.blender.org/blender/blender/pulls/127784
More thoroughly test certain image processing behavior of USDZ IO.
Namely that regular and UDIM images are imported back when using either
the packed or the copy import modes. Along the way ensure that we have
coverage for image downscaling on export as well.
Pull Request: https://projects.blender.org/blender/blender/pulls/127985
Export Blender Point Clouds as `UsdGeomPoints` primitives.
Summary
- Adds the `USDPointsWriter` class
- Adds tests to ensure that animated positions, velocities, radii, and
regular attributes are all written correctly (in a sparse format)
- Adds a new `export_points` operator property, mirroring the existing
`import_points` option
Pull Request: https://projects.blender.org/blender/blender/pulls/126389
Newly validates the following:
- Image mapping transforms on import and export
- Typical normal map setups on import and export
- Alpha-clip node setups on import (was already tested for export)
Pull Request: https://projects.blender.org/blender/blender/pulls/126776
Additional coverage for the following scenarios:
- Ensures custom properties are exported and imported correctly
- Ensures that xform op modes and scene orientation options are properly
respected during export
Pull Request: https://projects.blender.org/blender/blender/pulls/126723
Verifies all currently supported USD Shape prim types are successfully
imported. Each prim has time sample data authored so the presence of a
Mesh Sequence Cache modifier is also checked.
Pull Request: https://projects.blender.org/blender/blender/pulls/126449
The mesh velocity data was not using the UsdUtilsSparseValueWriter and
was writing out data for all frames even if the velocity didn't change.
Adds test coverage for this scenario as well as other situations where a
MeshSequenceCache (MSC) would be required:
- Ensures that when positions vary a MSC is added
- Ensures that when velocities vary a MSC is added (see blender/blender@c862d40e09)
- Ensures that when attributes vary a MSC is added (see blender/blender@3c394d39f2)
Pull Request: https://projects.blender.org/blender/blender/pulls/126208
Adds test coverage for a few, minor, missing cases for USD Light export:
- Spot lights with the special case of a 0 radius
- Area lights of type SQUARE (RECTANGLE was already tested)
- Area lights of type ELLIPSE (DISK was already tested)
Pull Request: https://projects.blender.org/blender/blender/pulls/126203
Add support for reading and writing custom `Curves` attributes.
This allows us to roundtrip Blender's Hair grooms containing UVs and
other attribute data. Note that animated attribute values are not
supported with this change.
This will also address #120042
Pull Request: https://projects.blender.org/blender/blender/pulls/121928
Adds a test to validate the import of Mesh attributes for USD.
It reuses the recently added export attribute test file to generate the
USD file to load.
The test validates the current behavior in `main`. Missing data and bugs
are noted in comments and can be addressed afterwards.
Pull Request: https://projects.blender.org/blender/blender/pulls/121327
The curve variations were used during development of the GeometrySet
changes.
The lights and point instancer coverage would have helped uncover bugs
earlier. Bugs that eventually had to be fixed in 4.1. Better late than
never.
The one downside is that the light tests is actually a round-trip test,
which is normally fine to do, except it technically does an export
during the import test.
Pull Request: https://projects.blender.org/blender/blender/pulls/119858
Calling `WM_report` & co API from wmJob worker thread is utterly unsafe,
and should never have been done. It 'worked' so far presumably because
worker threads were barely (if ever) reporting anything that way, but
now USD IO code is spamming reports in some cases, leading to fairly
common crashes.
Pull Request: https://projects.blender.org/blender/blender/pulls/113883
Fix various issues in import and export of cameras.
* Wrong units and flipped shift values on import
* Wrong clip start lower range on import
* Wrong units for aperture and focal length properties on export
* Issue with auto sensor fit and height > width on export
Pull Request: https://projects.blender.org/blender/blender/pulls/112905
Added support for UsdSkel animation import.
This addresses #110076.
Added USDSkeletonReader class which imports UsdSkelSkeleton primitives
as armatures.
Extended USDMeshReader to import UsdSkelBlendShape as shape keys.
Extended USDMeshReader to import USD skinning data as as deform groups
and an armature modifier on the mesh object.
Added USDMeshReader::get_local_usd_xform() to override the transform
computation to account for the binding transformation for skinned meshes.
Pull Request: https://projects.blender.org/blender/blender/pulls/110912
Listing the "Blender Foundation" as copyright holder implied the Blender
Foundation holds copyright to files which may include work from many
developers.
While keeping copyright on headers makes sense for isolated libraries,
Blender's own code may be refactored or moved between files in a way
that makes the per file copyright holders less meaningful.
Copyright references to the "Blender Foundation" have been replaced with
"Blender Authors", with the exception of `./extern/` since these this
contains libraries which are more isolated, any changed to license
headers there can be handled on a case-by-case basis.
Some directories in `./intern/` have also been excluded:
- `./intern/cycles/` it's own `AUTHORS` file is planned.
- `./intern/opensubdiv/`.
An "AUTHORS" file has been added, using the chromium projects authors
file as a template.
Design task: #110784
Ref !110783.
Follow connections when reading the varname attribute of a primvar
reader, and support both string and TfToken types for the varname.
A unit test is also provided.
Authored by Apple: Matt McLin
Pull Request: https://projects.blender.org/blender/blender/pulls/105508
Authored by Sonny Campbell.
Currently when importing a USD file, some of the camera properties are
ignored, or the units are not converted correctly from USD world units.
On import we currently set the focal length, but not the camera sensor
size (horizontal and vertical aperture), so the camera field of view
is wrong. The sensor size information is in the USD file, but is ignored
for perspective cameras.
USD uses "tenth of a world unit" scale for some physical camera properties
like focal length and aperture.
https://graphics.pixar.com/usd/release/api/class_usd_geom_camera.html#UsdGeom_CameraUnits
I have added the UsdStage's metersPerUnit parameter to the ImportSettings
so the camera can do the required conversion on import. This will convert from
the USD file's world units to millimeters for Blender's camera settings.
Reviewed by: Sybren and makowalski.
Differential Revision: https://developer.blender.org/D16019
