Advice on relationship targets to prims in external layers needed

Community USD
1

Hi! I work at Remedy Entertainment where we’re using USD as our level data format. We’re currently trying to add support for something we call (for lack of a better name) ‘External References’ and I’ve come here looking for some advice as what we tried so far hasn’t really worked.

The way we’ve structured our layers is that we have a top-level layer per level, a bunch of other layers for composable things like areas, character, items, and so on, and then we also have a layer for global data which we want to point to from various layers. (This is simplified but I think will help get the point across)

We export our levels and global data to a runtime format where we identify objects by an id that is essentially a hash of its usd path. During runtime our global data is always loaded and we might have one or more levels loaded as well at the same time.

Now, up until this point, whenever a layer needs to point to any of the global data we’ve done so by manually entering string ids for the data you want to reference. We would however much prefer to point to it via relationship targets (to make it work with other tooling we have) and convert those targets into our ids. The problem we have is that because the id is based on the object’s usd path we can’t bring the global data layer in as a usd reference because that of course changes the object’s path.

So we tried instead to sublayer the global data in. But then we stumbled upon another problem which is that the relationship targets don’t seem to survive the reference composition arc. Let me give a simplified example.

We have items.usda:

#usda 1.0
(
    defaultPrim = "items"
)

def "items"
{
    def "briefcase"
    {
    }
}

player.usda:

#usda 1.0
(
    defaultPrim = "player"
    subLayers = [@items.usda@]
)

def "player"
{
    rel equipped_item = </items/briefcase> 
}

level.usda:

#usda 1.0
(
    defaultPrim = "level"
    subLayers = [@items.usda@]
)

def "level"
{
    def "player" (
        references = @player.usda@
    )
    {
    }
}

If we run the following code:

auto stage = pxr::UsdStage::Open( "level.usda" );

pxr::SdfPathVector items;
stage->GetRelationshipAtPath( pxr::SdfPath( "/level/player.equipped_item" ) ).GetTargets( &items );

Then we get the following warning:

In </level/player.equipped_item>: The relationship target </items/briefcase> from </player.equipped_item> in layer @player.usda@ refers to a path outside the scope of the reference from </level/player>.  Ignoring. (getting targets for relationship </level/player.equipped_item> on stage @level.usda@)

And items is empty.

That makes sense (at least I think so). When composing </player.equipped_item> into </level/player.equipped_item> it doesn’t know that level.usda also sublayers in items.usda and that the path is therefore valid.

Since sublayering didn’t work, our next idea is to bring the global data layer in as usd references but do some sort of re-mapping scheme in our tooling when we export the paths into our ids for our runtime format. But is there a way to make the sublayering work? Or are there any other ideas or advice on how we could structure our data to get the sort of referencing we’re looking for?

2

I haven’t really thought this through, but have you looked at using “inherits” to get at your global data? Inherit (and specialize) composition arcs have the unique behavior of reaching “outside” the reference to the matching location on the composed stage. So any layer that needs to access global data could have a prim “/locals” that “inherits” from “/globals”. within this layer, you could have your relationships point to “/locals”, which would be empty normally (though who knows, maybe you would want to put some layer-local data there too).

Then on the full stage, you’d either sublayer in the layer with the “/globals” prim with the global settings. Or maybe you’d reference the settings onto “/globals”? I still can’t do composition in my head so I’m not totally sure which (if either) of these approaches would work, but I bet you could come up with something. Then all the layers with relationships pointing to “/locals” would see all the local settings with the “/globals” settings overlaid on top as stronger opinions.

Would that get you what you’re looking for?

3

Hmm, I’m definitely not an expert on inherits/specializes (we don’t use them at Remedy (yet)) but I don’t think that would do much for the problem.

If I’ve understood you correctly, my previous example would then look something like this? (Note: I renamed items.usda to global.usda)

player.usda:

#usda 1.0
(
    defaultPrim = "player"
)

def "local" (
    inherits = </global>
)
{
}

def "player"
{
    rel equipped_item = </local/briefcase> 
}

level.usda:

#usda 1.0
(
    defaultPrim = "level"
)

# alternatively, this could be brought in as a sublayer
def "global" ( 
    references = @global.usda@ 
)
{
}

def "level"
{
    def "player" (
        references = @player.usda@
    )
    {
    }
}

Unless I’ve misunderstood then I think this has the same problem as adding global.usda as a sublayer to both player.usda and level.usda i.e. that </player.equipped_item> won’t compose through the references arc. Also since the path to the prim we’re targeting in the relationship is different we’d still have to do some kind of re-mapping. I.e. we can’t just hash the path in the relationship since it won’t be the same as the path to the briefcase in global.usda.

One thing I feel I should clarify is that our primary concern isn’t being able to read the data composed from global.usda in the level.usda stage. What we’re mainly interested in is a way for the absolute path that the relationship is composed to in level.usda to be the same absolute path as if the relationship was authored & composed in global.usda. Reading the opinions in global.usda from the level.usda stage would be nice, but mainly for validating that the relationship target(s) targeting a prim in global.usda exists. We would ideally want to prevent users from authoring local opinions in level.usda on the global data.

4

Hi @morris , achieving that stability of target-path through one-or-more levels of referencing just isn’t something USD can provide. Like @mtucker , I thought there might be a trick involving the way that global inherits map through referencing into the global namespace, but it turns out there isn’t (which is probably good).

The way we deal with character items/props is by building an assembly for the character and referencing in each prop as a sibling of the character. You could also embed them inside the character - here’s an example of what that might look like in your player.usda:

(
    defaultPrim = "player"
)

def "player"
{
    # You could instead reference </items> here, if there aren't that many,
    # but then the individual items can't be instanced
    def "items"
    {
        def "briefcase" (
            references = @items.usda@</items/briefcase>
            instanceable = true
        )
        {
        }
    }
    rel equipped_item = </player/items/briefcase> 
}

That target path will obviously be different/translated in level.usda (which no longer subLayers items.usda), so does not give you what you want. But the fact that we are instancing the items (in addition to producing more compact scene description), gives your code a fairly easy way to identify all the sited that are using the same item. And also because only instanced versions of the items are present in level.usda stage, users would not be able to override anything about the items, beyond what’s described on their root prims.

Without know more about why you want the target paths to be identical in both asset and scene, I’m not sure what else to suggest.

Cheers,
spiff

5

Thanks for the reply! I got busy at work with some other stuff but I’m now back on this.

That is the conclusion we came to as well so we’re looking into a solution similar to your example. Something like this:

def "player"
{    
    class "items" (
        instanceable = true,
        references = @items.usda@
        externalReference = true
    )
    {
    }

    rel equipped_item = </player/items/briefcase>
}

Some notable differences:

Class
I’ve only ever seen class prims in examples when used with inherit/specialize but the class entry in the glossary seems to indicate that that’s just a common use of specifying prims as classes and not its primary effect which is that the prim and its descendants are flagged as abstract.

In our case we’re not interested in visualizing the external references in our viewport (or really any tool’s viewport, be it ours or a DCC), we just want their data to be resolvable in the Stage for our tooling. So we make them abstract so that default traversal of the stage doesn’t include them.

That to me sounds like what Abstract prims were meant for but I wonder if it’s actually standard practice in USD to use classes in this way?

Referencing the default prim
We expect that we’ll have a lot of relationships that target multiple prims in the same layer. If each of those targets were a separate reference prim then there’s a lot more to manage when a user changes it from one target to another.

For instance if the equipped_item was changed from a briefcase to a flashlight then we’d have to remove the briefcase reference prim and add the flashlight reference prim. While if we have a single reference prim for the default prim it’s only a matter of modifying the relationship.

We don’t have any good tooling around this issue (and we haven’t figured out any good ways to deal with it either) so our current plan is to just have users manage it themselves and so minimizing the amount of reference prims they need to add/delete would be beneficial. We’ve come across this wall a few times in other scenarios too so I’m curious if there are any solutions out there anyone has come up with?

Another reason to not reference the target prims is that it’d create more prims you could potentially author overrides on since the instanceable prims can still be overridden.

The obvious downside of all this is of course that we’re potentially bringing in prims we don’t need.

externalReference metadatum
This is so we can figure out if our target is pointing to an externalReference and we need to re-map it.

Here is another downside of referencing the default prim instead of the target though. To figure out if our target path needs to be re-mapped we’ll need to check its ancestor prims if it has the metadatum. Which is pretty expensive to do for all of our relationship targets. We can early out if the prim isn’t abstract (which isn’t a very expensive check) but likely we’ll implement some custom cache of this metadatum to speed this up since we could benefit from this in other places too.

6

Also wanted to mention, when we first started looking for solutions to this problem of ours the first thing we looked for was to a relationship target in the style of a reference. E.g. rel equipped_item = @items.usda@</items/briefcase>

I still think it would be ideal if that was somehow possible in USD since it both feels intuitive once you’ve worked with references and it is exactly what we’d need. I.e. an absolute path to a prim in another layer that you can’t override.

I don’t doubt that it wouldn’t be trivial to implement though :stuck_out_tongue:

7

At Pixar we generally reserve “class” specifier for inherits-based overrides so that it’s easier for GUI’s to find such containers, but that’s not a rule, and it’s only even actionable for root prims, and you’re using it inside a mode, so go for it. Abstract is a much nicer “protector” of prototype or other data than “over” is.

Given that you are instancing the default prim, I wouldn’t worry too much about the cost of the extra prims you are bringing in, as it will be amortized over all your players.

Not sure I have any thoughts on your externalReference metadata problem.

I will throw out one other crazy idea… structure your items.usda such that the default prim creates a variantSet for each item (e.g. “includeBriefcase”). The “on” or “include” variant of each of those sets would both reference (on a unique child prim… reference could be instanced, also?) the appropriate item and add an override to the equipped_item rel to target the same item. Crucially, you now reference items.usda onthe player root prim, not inside it

Now, selecting items involves only making a variantSet selection, and only the items you use are populated, per-player. The downside is you potentially have a ton of variantSets on those player roots. This is actually similar to how we structure our “character groups”, though we tend to have only a small set of configurations, so it’s much simpler a setup involving a single variantSet.