Building An Asset Pipeline

[Raymond]

We plan to create a system that pre-creates all our asset-layers up front, like the structure explained below.

Not all assets will need all of these layers , if the layer is not needed the artist will just leave the layer empty. The prims in the pre-built layers will be deactivated, so nothing would really be loaded from the layers until an artist publishes something to the layer.

We plan to use this same “asset” structure for building character, props and sets in addition to smaller components like an apple, or a coffee mug.

The reason we want to do this is, we don’t have to decide what layers will be needed at the point of creating the asset/component. If we find out at a late stage that we need to do some groom on the asset, the artist can open the work file, and publish to the groom layer.

Or if we find that all we actually need for a specific asset is assembly, we could just leave all the other layers blank, and just work on the assembly layer.

To us this makes sense, but I wanted your thoughts on this before we start implementing it.

• Are others doing something similar to this?
• Do you separate components and assets and consider them to be different “things” in your pipeline?
  • If so, why ?
• Do you see any pitfalls?
• We want to build our pipeline as close to a standardized usd asset workflow as possible, do you see this is within the standard?

Payloads

When do you use payload, do you use it for all layers no matter the complexity of the content?

Animation

For assets that require a complicated rig like a character, a normal maya rig is still the way to go, but for simpler assets that only need xforms, do you bring in the usd asset into your animation scene, or are you still using maya rigs for these assets?

A Lot of the concepts of USD can be a bit abstract until you “get it", we had a group internal at our studio doing several really inspiring and interesting meetings about our implementation, it has taken a some time for us to all get on the same page and grasp all the concepts and ideas of usd. We are not fully sure we get it all, and are going about this in the correct way, this is the reason we wanted to get some input from others that have been through a similar process.

Thanks

[spiff (F Sebastian Grassia)]

Great questions, @Raymond, and welcome to the forum! Before sharing a Pixar perspective, I have a few questions for you:

• Can you clarify whether AssetName.usd contains five subLayers, or directly encodes five VariantSets?
• When you talk about a publish step for an asset, is there a distinction between “source” and “installed”, with a “asset build process” moving from source to the versioned-inst, or is “publish” just checking a file in?
• When you ask “Do you separate components and assets and consider them to be different “things” in your pipeline?”, do you mean: as opposed to each layer comprising a component being an independently rc’d asset?
• Can you explain what goes into an asset-level set-dress layer? I think I’m understanding that the assembly layer would only ever be active for aggregate assets (i.e. sets/environments), but if that’s so, then the only thing I can mentally map setdress to is sequence/camera-specific edits that would never belong in any asset, but rather in a sequence or shot layer.

[Raymond]

Hi Spiff, thank you for the warm welcome, happy to be part of this new community

> Can you clarify whether AssetName.usd contains five subLayers, or directly encodes five VariantSets?

Our current plan is to set it up something like this,

#usda 1.0
# AssetName.usd
# This is just a sketch, so there might me  typos and errors 

(
    defaultPrim = "assetName"
    metersPerUnit = 1.0
    upAxis = "Y"
)

class "__class__"
{
    class "assetName" {}
}

def Xform "assetName" 
(
    inherits = </__class__/assetName>
    kind = "component"
    add variantSets = ["model","setdress","groom","look"]

   
    variantSet "model" = {
        "default" (payload = @./model/pub/latest/assetName-default-model.usd) { }
        "broken" (payload = @./model/pub/latest/assetName-broken-model.usd) { }
    }

    variantSet "setdress" = {
        "default" (payload = @./setdress/pub/latest/assetName-default-setdress.usd) { }
        "broken" (payload = @./setdress/pub/latest/assetName-broken-setdress.usd) { }
    }

    variantSet "groom" = {
        "default" (payload = @./groom/pub/latest/assetName-default-groom.usd) { }
    }

    variantSet "look" = {
        "default" (payload = @./look/pub/latest/assetName-default-look.usd) { }
        "broken" (payload = @./look/pub/latest/assetName-broken-look.usd) { }
        "wet" (payload = @./look/pub/latest/assetName-wet-look.usd) { }
    }

    prepend references = @./assembly/pub/latest/assetName-broken-assembly.usd { }
    
)
{ }

Looking at this now i think we will need to update the AssetName.usd file dynamically so we can add new variants when they are published.

• When you talk about a publish step for an asset, is there a distinction between “source” and “installed”, with a “asset build process” moving from source to the versioned-inst, or is “publish” just checking a file in?

Yes there’s a distinction between the two, dev would a seperate folder where the artist work, and then they will publish their work to pub folder containing one version for each publish.

/dev
     /assetName-mode.ma ( Source)
/pub
     /v001
          /assetname-model.usd 
     /v002
          /assetname-model.usd
     /latest
          /assetname-model.usd
  

.

• When you ask “Do you separate components and assets and consider them to be different “things” in your pipeline?”, do you mean: as opposed to each layer comprising a component being an independently rc’d asset?

Not sure if I understand the term rc’d asset ?

My question is a bit vague, and i guess thats because im confused about how it’s used in example files and how it is referenced to as A Component is the most basic asset published in a USD pipeline
In that documentation is referred to a basic asset, does that mean that a complicated asset would be considered to be something different and the two needs to distinguished from eachother in some way?

Also the houdini component-builder seems to be built with simple assets in mind, i would probably not use that for building my hero character. ( i might be wrong )

• Can you explain what goes into an asset-level set-dress layer? I think I’m understanding that the assembly layer would only ever be active for aggregate assets (i.e. sets/environments), but if that’s so, then the only thing I can mentally map setdress to is sequence/camera-specific edits that would never belong in any asset, but rather in a sequence or shot layer.

In our current pipeline, asset-level setdressing is where we scatter trees, rocks etc onto a landscape, or placing cups, plants, kettles and so on into a messy kitchen asset, we dont got a assembly step in our current pipeline.

Assembly in this new suggested pipeline, we thought of being used more for putting together bigger assets to create new assets. For Example if HouseA, HouseB and HouseC is separate Asset, Assembly would be used to bring in these assets and place them to form a new asset .

As i’m writing this, i’m thinking Setdress and Assembly actually could be the same thing. I am curious about your thoughts on this, in what asset-level layer do you do what we call setdressing (scattering grass, placing trees, rocks and so on)

Thank you for helping us out with these questions

[flord (Francois Lord)]

The way I see it is like this: components assets are simple assets, assembly assets gather component assets. So assembly assets declare a hierarchy which has one single parent, and then possible multiple Xforms of kind “group”. Under the Xforms you could find prims which reference some components assets. You can also have instancer prims in there to create more complex layouts. In that sense, assembly assets are more similar to shots in their structure than to component assets. You could even have animation published for the assembly asset. For example, you have a landscape with wind mills and you want the wind mills to have the same animation in all the shots, but each mill has it’s own animation. You publish the animation bake at the assembly level, just like you would at the shot level or sequence level.
So to answer your question

@Raymond

Do you separate components and assets and consider them to be different “things” in your pipeline?

I assume you mean component and assemblies. The answer is yes. Assemblies are flagged differently in the asset management system and the pipeline behaves differently for them. Both of them are still assets, though. They both need one single parent (root prim) in their hierarchy, and they both can be referenced in shots.
There is also the topic of “can I publish geometry directly inside an assembly or do I need to create a component asset for every single thing I need in my assembly?”. You decide!
We ended up allowing publishing geometry inside assemblies for things like the ground geometry in environments. We were tired of having components assets that were used only in one assembly asset. We used the same mecanism allowing artists to publish geometry to shots for things like shadow casters and receivers which are common in VFX. So again, assemblies are similar to shots.

[spiff (F Sebastian Grassia)]

We actually tried something similar to this in the years leading up to USD’s development in Presto’s scene description, which is very similar to USD’s. It was called “the unified build”, and the goal was to make it much easier to combine different asset feature-pipelines together in a single asset… like a furry blanket, or a rigged hat. And I think it accomplished those goals; but the cost was that it made every single asset uniformly more complicated and heavy… and the assets that benefitted from the ease-of-combination were typically 10-15% of total assets.

We wound up with something more complicated than what you are proposing, and one of our motivating examples for the new USD pipeline we built as we deployed USD was that the number of layers needed to define a single-Mesh spoon asset drop from around 12 to 3. Speaking for just myself and not Pixar, it seems like “full instantiated template” asset structure can be nice for asset authoring, but when trying to “read” an asset, i.e. use, understand, find out where problems might be, there’s alot to be gained from making every asset as simple as it can be.

So that’s why I asked if you have a (scriptable) publish step for your assets, where you could remove all the complexity that the asset-creator(s) have not at that point utilized. We’ve actually gone on a similar but “from the other direction” path, where every asset starts out uniformly simple*, but we’ve made it easy for artists to add layers to incorporate work from various other tools, and have several “build steps” that look for certain tags to add particular feature packages in.

That’s just one perspective, and your mileage may definitely vary!

• That’s not entirely true, since Presto-rigged assets follow a fairly significant fork in the asset pipeline, and we generally identify those assets up-front, I think.

[Raymond]

A separate layer for the anim department to animate at an asset level is a really cool idea, we might just copy this idea into our pipeline flord

There is also the topic of “can I publish geometry directly inside an assembly or do I need to create a component asset for every single thing I need in my assembly?”.

We have been discussing this topic internally as well, I think we are also going to allow creating geometry inside an assembly. I imagen frustrated artists if they have to create new assets for things that are very asset specific and are not going to be used anywhere else. We might want to make sure the additional geometry ends up in a separate layer.

Layers

When it comes to pre-building all layers, I think I got a much clearer picture of how we want to build this now, Thanks.

Instead of rebuilding all the layers, I think we will only pre build the model and look layer, these layers are common between all asset types so I think it makes sense to pre-build these.

If other layers are needed for an asset, the artist would just reopen the build asset GUI, and add the new asset steps that they want. This in turn will create the workfiles into the correct folder structure .

When publishing, we will dynamically rebuild the asset.usd file with the newly published layer.

at least this is the current plan, but things might change

Payload is still a bit unclear to me , do you only add payloads if the assets are beyond a complexity threshold, or do you use payloads for all assets no matter the complexity.

[spiff (F Sebastian Grassia)]

Payloads protect interactive performance on two levels, if you do leverage load-state in your workflows.

First, it guards stage-opening performance (and memory footprint), by an amount that depends on number of prims, number of layers, and number of composition arcs in a payloaded asset. We don’t have a precise formula for this, but you might be able to build an intuition for this that could be codified in a build script.

But the second gate payloads provide guards against client-processing, e.g. viewport rendering, and I think it may be more difficult to come up with a good heuristic for this. As an exmple, you could have a compositionally trivial asset with a single Mesh and Material, comprising just a dozen prims. But the mesh could have a million verts, have timeSampled animation applied, and a material with displacement. If you scatter that thing with LayerOffsets so that the animation is offset on the different instances, that’s a whole lot of data bandwidth and computation to render.

We do add a payload to every component asset, which is why our initial structure was three layers instead of two (you can see this in the KitchenSet example, but not on assembly assets. The exception for assemblies is so that if you open a stage fully unloaded, you can still immediately see the entire “model hierarchy”… and also because it took several years before we enabled “non-recursive payload inclusion” (i.e. UsdLoadWithoutDescendants).

[Raymond]

Thanks, this is very helpfull.

Our pipeline is progressing realy nicely, thanks for all the help