Copy in memory stage and its layers to be unique copies

Community USD
1

In USD Python is there any (easy) way to take the USD stage and layer instances I may have and make a copy in memory of them?

I’m trying to keep a copy of a Houdini LOP node’s stage and layers in memory, and requery it again with different context option. However, since both share the same USD stage and Sdf layers in memory it means that cooking again basically seems to have updated the initial instance too.

# simplified pseudocode
lopnode: hou.LopNode
hou.setContextOption("shot", "sh010")
sh010 = lopnode.stage()
hou.setContextOption("shot", "sh020")
sh020 = lopnode.stage()

I want to be able to access both of the stages in the state of those context options.

I guess I could technically do something like:

stage: Usd.Stage
uuid = str(uuid.UUID())
root_layer = stage.GetRootLayer()
root_layer_clone = Sdf.Layer.CreateInMemory(root_layer.identifier + uuid)
root_layer.TransferContent(root_layer_clone)
for layer in stage.GetLayerStack():
    clone = Sdf.Layer.CreateInMemory(layer.identifier + uuid)
    layer.TransferContent(clone)

# But then I'd need to also remap ALL the layers from their original references to the new cloned in memory references?
# Plus I'd essentially only really want to clone the in-memory layers then that in this case are relevant to Houdini's graph (but that's maybe an optimization only)

clone_stage = Usd.Stage(root_layer_clone)

However, I’m not sure how to easily remap them correctly to their cloned once.

But even better, I’m sure I’m being stupid here and there must be a better way!

2

I ended up doing this for now:

def copy_stage_layers(stage) -> dict[Sdf.Layer, Sdf.Layer]:
    # Create a mapping from original layers to their copies
    layer_mapping = {}

    # Copy each layer
    for layer in stage.GetLayerStack(includeSessionLayers=False):
        # It seems layer.IsAnonymous() fails (does not exist yet?) so we use
        # the identifier to check if it is an anonymous layer
        if not Sdf.Layer.IsAnonymousLayerIdentifier(layer.identifier):
            # We disregard non-anonmyous layers for replacing and assume
            # they are static enough for our use case.
            continue

        # Sdf.Layer.TransferContent seems to crash, so instead we export
        # and import (serialize/deserialize) to make a unique copy.
        layer_str = layer.ExportToString()
        copied_layer = Sdf.Layer.CreateAnonymous()
        copied_layer.ImportFromString(layer_str)
        layer_mapping[layer] = copied_layer

    # Remap all used layers in the root layer
    # TODO: Confirm whether this is technically sufficient?
    copied_root_layer = layer_mapping[stage.GetRootLayer()]
    for old_layer, new_layer in layer_mapping.items():
        copied_root_layer.UpdateCompositionAssetDependency(
            old_layer.identifier,
            new_layer.identifier
        )

    return layer_mapping

And using it along the lines of:

copied_layer_mapping = copy_stage_layers(stage)
copied_stage = Usd.Stage.Open(
    copied_layer_mapping[stage.GetRootLayer()])
copied_layers = [
    # Remap layers only that were remapped (anonymous layers
    # only). If the layer was not remapped, then use the
    # original
    copied_layer_mapping.get(layer, layer) for layer in layers
]

Which seemed to work, without crashing and allowed me to access the stage and layers in the state I wanted to down the line.

For context, example usage in this PR: Support Houdini context options from the ROP node for the USD collectors + validations by BigRoy · Pull Request #145 · ynput/ayon-houdini · GitHub

Better ways?

Whether it’s bad practice I’m not sure - so input would be greatly appreciated!

3

Hi @BigRoyNL ,
Firstly, you’re not being stupid and you didn’t miss anything. “Cloning” a UsdStage could mean alot of things, and tackling it in its full generality (which is what we’d likely want/need to do for the distro), would take some thought.
Secondly, it’s disturbing that TransferContent() is crashing on you… if you can provide a repro to SideFX, we’d be happy to work with them, if you can’t provide one directly to us on GitHub.
Thirdly, I’m surprised IsAnonymous() is not functioning correctly for you; the layers must be fully established when you get them, so that’s not it. A repro on that would be great, also.

Finally, as to what you are doing being sufficient:

  1. It is unlikely, but possible for the sublayers of the root layer to refer to other layers in the layerStack, so you want to do the UpdateCompositionAssetDependency() loop on all your new layers.
  2. Also (possibly even more?) unlikely in the Solaris workflow, but UpdateCompositionAssetDependency() will not update any asset valued attributes or metadata (other than composition arcs)

Addressing #1 is easy. Addressing #2 would require a different pattern, like using SdfCopySpec() to do the copying (with a callback to modify asset paths) to do the copying in the first place, or Using SdfLayer::Traverse() after you’ve done the copying. Not sure if this is likely enough to be worth your effort…

What you’re trying to do is close to what we do in the asset localization code in UsdUtils, and an enterprising developer could provide a new front end for cloning that leverages the same guts to do it robustly.

Cheers,
–spiff

1 Like
4

Thanks so much for the quick response @spiff

Those comments were there purely for myself - and weren’t necessarily reporting a bug. Nonetheless good you spotted them and thanks for mentioning how to approach from here.

I can’t seem to reproduce this crash now. I thought I initially at the transfer order wrong: layer.TransferContent(copied_layer) instead of what it should be: copied_layer.TransferContent(layer). But even then I can’t seem to get it to crash again (and it gives me an error that I’m not allowed to transfer into the LOP managed layers). So I’m not sure how I did that initially.

So ignore for now :white_check_mark:

This was just me being stupid for a bit. The Python API exposes it as layer.anonymous attribute and not a layer.IsAnonymous() method call which I just happened to forget in that moment.

My bad. :white_check_mark:

It is unlikely, but possible for the sublayers of the root layer to refer to other layers in the layerStack, so you want to do the UpdateCompositionAssetDependency() loop on all your new layers.

Yes - valid point. I guess stage.GetUsedLayers() wouldn’t suffice either because it’d exclude those that are present outside of the composed stage itself?

Also (possibly even more?) unlikely in the Solaris workflow, but UpdateCompositionAssetDependency() will not update any asset valued attributes or metadata (other than composition arcs)

I see how that could be problematic. However, in this case I’m not actually swapping content - I’m only updating them to a copy of itself so those should still remain the same - or at least that’s what I’d expect.

I figured something close to this may have lived somewhere. I did take a peak at that one for a second but figured it may also do all actual asset paths (textures, etc.) which I wouldn’t care about, I’d only care about the files affecting the composing of the USD Stage (or its layers) but not necessarily things like textures.

Another quick peek now does seem like the actual redirecting/localizing of the files instead of cloning is quite embedded into its code that it wouldn’t be a 1:1 straight copy.

For anyone else looking to take a ‘quick peek’ at that: UsdUtils asset localization code is here.

I also thought that maybe supplying a “user processing function” could have allowed me to customize the behavior to only remap layers as clones but it felt overkill and I wasn’t sure how to query whether the path was influencing the USD composition (e.g. is a valid layer or is just a resource like a texture) so I refrained from going that route.

5

You should probably be submitting a question to SideFX support about this. I strongly suspect that there are more efficient, more node-centric ways of accomplishing whatever you’re trying to do here. In particular, I suspect there are ways of forcing LOPs to make these copied stages you need, and LOP nodes will do it in C++ in a way that is as efficient as we can be knowing everything we know about how LOP stages are constructed, what layers in the network are actually changed by the change to the context option values, etc.

By trying to implement this as a general or even semi-general algorithm in python I think you’re missing out on a lot of potential optimizations, not to mention writing a lot more code than you should need to.