Complex attribute data in USD

Community USD
1

Dear AOUSD folks,

In this week’s USD Games Working Group meeting, we’ve been discussing how to describe complex attribute data in USD. Complex attributes are those encapsulated in structures. They may also be formatted as arrays and dictionaries and even nested within each other.

These complex attributes are common in the game industry. For instance, in Unreal Engine, a static mesh asset has a structure attribute named Nanite Settings, which encloses the configurations for the Nanite system. It also has a structure array attribute named Building Settings that encloses the typology information for each level of detail.

This year in GTC, cooperating with NVIDIA, we’ve been sharing our solution used in Tencent Games for the past couple of years. The record has been released to: Build Games in the Cloud: How Tencent Streamlines 3D Content Production With OpenUSD | NVIDIA On-Demand

image
image1920×1080 139 KB

As shown in the above slide, our solution is extremely simple and based on namespace. However, it cannot support structure arrays. We’ve been thinking of flattening structure arrays into array attributes, but it hurts the readability of the USD stage, and it also does not support nested structure arrays.

It’s not just us who are struggling with this problem. Jeremiah Zanin and Levi Biasco of Santa Monica Studio have faced the same challenge and ended up with a comprehensive solution named “Compound Attribute.” Here is the record of their presentation: Video Conferencing, Web Conferencing, Webinars, Screen Sharing - Zoom

image
image1920×1084 89.3 KB

Compound Attributes can describe the most complex data, whether they are structures, structure arrays, or nested arrays. They have used it wildly in production. However, this solution buries a performance pitfall since it leads to an exponential increase in USD attributes.

Having an array of one million elements in USD is fine, but having a million attributes is totally another story. We’ve been testing it and found it can cause significant performance issues. In practice, Jeremiah and Levi have not encountered such a complex stage, but it was difficult for us to decide and preach it as a standard for the entire USD community.

So far, there is no common agreement on describing complex attributes. So, I wrote this article and am looking for help from the community. Has anyone encountered a similar problem at work? What would your suggestion be? How about eliminating all these solutions by using a TfDictionary? We are seeking a solution that will benefit all game studios, and the final standard might be officially implemented in Unreal Engine to reflect properties as USD attributes.

Cheers,
Calvin Gu

1 Like
2

How linear is the performance cost of having lots of attributes? My reflex here is to come at this from a runtime optimization perspective, and see if there’s a reliable way to hash and/or elide namespaced attributes, but I’d need to pull together some benchmarks to get a better sense of how applicable that approach would be.

3

Thanks for the clear description of the problem, @CalvinGu . Concur that there is a fundamental difference in behavior between many-prim-siblings and many-property-siblings. In the Sdf data model, the two are pretty much identical, and suffer from non-linearities in authoring (adding N new prims or properties to a spec sequentially is basically N^2). On the reading side, it should be fast (insensitive to number of siblings) to query the existence of and Get() value of both composed UsdPrims and (in the latter case) UsdProperties.

Where the two diverge in read performance is enumerating siblings or subsets of siblings, because a UsdStage caches this result for prims, but for properties it must be computed through a PrimIndex every time. And the fact that we don’t cache property-data is a key scalability aspect of UsdStage, both for resident memory consumption, and latency in opening a UsdStage.

I would be very curious to hear about where the key performance bottlenecks show up for you. @Jean-Silas , we do already try to optimize when enumerating properties in a namespace, but there’s only so much we can do without considering some potentially painful changes.

So, from the other direction… if millions-of-properties did perform reasonably, and allowed you to leverage applied and multi-apply schemas for modeling compound structs, would there still be an argument that something like “dictionary-valued attributes” would still be the superior solution? If so, could we discuss?

4

Thanks for your explanation, Spiff!

Before discussing the difference between UsdPrim and UsdProperty, I would like to clarify further that both can cause performance pitfalls. The question I would like to bring up here is for those complex attributes: is it worth using multiple UsdPrim and UsdProperty, or should it be better to use a single UsdProperty?

UsdProperty does not support dictionary-type values. Moreover, even if it is supported, the current implementation of TfDictionary cannot describe the complex data we desire. The below snippet can tell:

self.assertEqual(
    Sdf.ConvertToValidMetadataDictionary({'a':['hello', 1]}),
    (False, {'a': None},
     "failed to cast array element 1: <int> '1' under key 'a' "
     "to <string>"))
self.assertEqual(
    Sdf.ConvertToValidMetadataDictionary({'a':[{}, {}]}),
    (False, {'a': None},
     "first vector/list element <VtDictionary> '{}' under key 'a' "
     "is not a valid scene description datatype"))
assertEqualish(
    self,
    Sdf.ConvertToValidMetadataDictionary({'a':[[1,2,3], [2,3,4]]}),
    (False, {'a': None},
     ("first vector/list element",
      "'[1, 2, 3]' under key 'a' is not a valid scene description "
      "datatype")))

In production, when describing a game level in USD, the common stage would be about thousands of UsdPrim, and they each hold a complex attribute. Either using properties in namespaces or compound schema, it can easily go up to over one million of UsdProperty. And if we can have “dictionary-valued attributes,” it will go down back to thousands, which is an apparent improvement.

In Tencent Games, we seldom compose a complex attribute. In other words, no matter how it has been described in USD, it is always treated as a single. Therefore, I believe moving forward to “dictionary-valued attributes” is good for us, but I don’t know whether @jjzanin-SMS and @lbiasco-sie need to compose their complex data.

Consequently, I do believe “dictionary-valued attributes” can still be a “superior” solution. I have this strong feeling also because we are using Omniverse’s Wire format most of the time, and it can be more than three times slower than USDZ when it has over one million attributes. @LouRohan for visibility.

5

@CalvinGu do you need time varying values for these cases or would uniform variability suffice? In our pipeline we also have a scenario where the quantity of data is overly costly to encode as attributes, and metadata has worked pretty well for us. It is somewhat composable and so the main limitation (other than the TfDictionary restrictions you noted) is the requirement that the data not be time varying.

6

I somewhat have a feeling that if going “one property” route that we wouldn’t be supporting separate ‘opinions’ on the property anyway? And hence you’re basically at a point where it becomes alsmost just as reasonable to store your ‘custom data’ as a JSON blob (or whatever blob). Not saying that’s good practice, but I suppose the performance implications from more properties (or even dict-values) would be way less acceptible?

(However that may be hardly what is up for discussion here since that’s maybe not “standardizing it” sufficiently from a specification standpoint for complex data)

But maybe I’m not sure the implications are of three times slower since that makes quite a difference if it’s 1s times three or 15s times three - so it’s hard to see from the discussion what the size was of the production implications of any performance issues due to going either way?

7

@CalvinGu We probably wouldn’t need composition for compound attributes, though it’s too early for us to say at this point. Particularly in nested compounds I could see there being edge cases. Dictionary-valued attributes would be interesting, but I’m not sure if they fit our needs as we still need a way to describe the structure of compound data. Metadata or default values could potentially be utilized to get that structure, but it gets shaky with nested and array compounds.

8

Just to keep expectations grounded, with respect to some of the issues/limitations @CalvinGu noted… in a hypothetical future in which USD attributes could be VtDictionary-typed:

  • the dictionary scene description type (backed by VtDictionary) is a container of nested dictionaries and other legal scene description types, so:
  • no arrays of arrays (but a dictionary[] type, i.e. array of dictionary would be provided)
  • Arrays in the Sdf datamodel require uniform element-type… you could fudge a non-uniform-typed array with a dictionary[] wrapping your elements, but not sure how attractive that is
  • metadata dictionary data today does compose element-wise through the composition, and that would be true for attribute dictionaries also, though whether dictionary[] attributes do the same dictionary composition element-wise within the array is one of those questions we’d need to wrestle with if dictionary attributes turned out to be something worth pursuing.

@lbiasco-sie , I’d love to hear more about the structure that is lacking from a dictionary that you need to encode separately!

9

We don’t need time-varying for those complex attributes. I think using metadata is an acceptable workaround as long as TfDictionary supports forms like dictionary under list.

10

Hey Spiff, I would say that if TfDictionary can support a dictionary under a list and a list under a list, then it can satisfy most of our needs.

Regarding another topic, I am unsure if I got you correctly. Is there not much difference in performance between many-prim-siblings and many-property-siblings? If so, I think the compound solution should be better than what we are doing now. If the refactor of TfDictionary and USD attributes is not going to happen shortly, we may think of switching to compound. At least it solved the “a dictionary under a list” issue.

11

@CalvinGu , “list under list” is “array of arrays”, and that’s something the data model will not support (except via the same trick I mentioned in my last post, via an array of dictionaries, in that potential future).

Many-prim-siblings and many-property-siblings both behave poorly for authoring (prims even more so because that triggers UsdStage recomposition, whereas properties do not), however they share the same ameliorative today: drop to the Sdf-level API’s and use an SdfChangeBlock to batch-create the siblings; that still doesn’t get down to fully linear, but it’s pretty good for rock 'n roll.

On the reading side, many-prims is fundamentally better than many-properties for iterating over the siblings, because prim-siblings are cached on the UsdStage, while property siblings are not. For individual UsdPrim or UsdProperty lookup by SdfPath and querying, the two should be roughly equivalent, and roughly constant-time.

12

Thanks, Spiff and everyone:

After analyzing each option, for performance and versatility reasons, I decided to go with this way:

USD Example
USD Example661×744 47.5 KB

  1. Until Pixar supports dictionary and dictionary[] attributes, the best place to describe complex data is the custom data.
  2. Given that an "array of arrays" is not likely to be supported in USD, we are thinking of using [{index}] dictionaries to represent arrays. Because USD supports nested dictionaries, every possible complex data should be supported once we practice this workaround.
  3. Since the attribute’s value will always be empty, we will set its type to group and treat it as a placeholder. Again, it can be refactored after Pixar supports dictionary and dictionary[] attributes.

We will deprecate the namespace-based convention, describe both single-structure and array-structure attributes in the above way. The primary reason we made this decision was to save on performance and the fact that we don’t need to compose complex attributes. The compound schema that @lbiasco-sie proposed is also reasonable, but since they cannot open the source code shortly, it will be too much for us to develop everything over again.

@mattyjams How do you feel about this one? Technically, it can describe every UProperty in Unreal unless it contains ":" or "[]" in its name. If it really happens, we will raise an error message and continue.

13

Sorry for the late reply everyone, I’ve been out on vacation.

@spiff After thinking about it, a VtDictionary type would help in most cases. It wouldn’t work for an “uninitialized dynamic compound array”, so a dictionary[] that starts empty yet needs some predefined structure for new elements. The other thing it would lack is the ability to place metadata on nested dictionaries (for example, the DataCompiler class name). But both those cases could be worked around, like with complicated metadata on the base attribute.

@CalvinGu A colleague of mine asked up the chain about open sourcing the CompoundAttributeAPI, and it sounds like it is on table! However I wouldn’t hold your breath on us getting it out as there’s plenty of red tape to get through first.

14

Hey, @spiff , please allow me to ask a follow-up question here.
As we chose to use VtDictionary to describe complex data, we recognized that VtDictionary uses “:” as the syntax sugar to query the nested keys.

However, if we assign a dictionary to VtDictionary in Cpp or Python with “:” in its keys, it actually works! Like this:
image
And it shows up like this:

image
image732×128 13.7 KB

Our question is whether this operation is legal in OpenUSD. We are concerned that one future release of OpenUSD will no longer support this syntax, causing us to rewrite it all over again.

15

Hi @CalvinGu , VtDictionary and Sdf’s use of it are indeed pretty free with what can be contained in the string keys. A note of caution, though, I discovered the usda file format does not properly escape control characters (or any characters), so it is currently possible to create dictionary entries that will result in a layer that can’t be parsed back in. We’ll need to discuss how we want to address that, but for now I’d say avoid control characters.

But aside from that, yes you can use : in keys, but it will cause you problems when trying to use GetCustomDataByKey(). If you could possibly use | or . or some other character for your substructure in keys, then you wouldn’t have that issue.

16

Could this be addressed with a convention that each (sub) dictionary has a dictionary-valued key called “metadata”, or am I misunderstanding and the “metadata” needs to be applied once (e.g.) for all of the entries in a dictionary[] array-valued attribute?

17

Yeah, the metadata would be something that describes the whole dictionary[] value, so to make it work in nested cases I think the metadata would have to be a sibling key-value pair of the actual nested dictionary[] key. So something like this for compound arrays

dict[] compArrayAttr (
    # Informs the default value for each entry in the dict array.
    defaultEntry = {
        float num = 1.0
        bool state = true
    }
)

versus like this for nested compound arrays

dict compoundAttr = {
    dict[] nestedCompArray
    dict nestedCompArray:defaultEntry = { ... }
}

This could work, but it would almost definitely be awkward since the metadata is stored differently between the cases.

18

Hmmm… could you not nest the defaultEntry for nested compound arrays? So every dict or dict[] would have a defaultEntry, which, for any dict[] member, gets its own, dictionary-valued entry in there? Seems more uniform, e.g.

dict compoundAttr = {
    dict[] nestedCompArray
    dict defaultEntry = { 
        dict nestedCompArray = {... }
        # Any other non-array fallbacks
    }
}
19

And also, one last question to aid our discussions… especially given this last explanation of the metadata, it really seems like we’re thinking about the structures as essentially schemas, that have fallbacks, etc.

So in figuring out how to think about the core problem here, I’m wondering whether if we could address the performance issues with having many properties on a prim (big big if), and we then added convenience API onUsdPrim or UsdSchemaBase to address applied multi-apply schemas as associative arrays (or maybe even index based with some caveats), would that be an intuitive way to model the data? Multi-apply (and single-apply) schemas can be effectively nested, so you can create structs of structs and structs of arrays-of-structs - though the latter would be fixed-length arrays for all nested schemas (which could be codeless). And because they are actual schemas, you get fallbacks from the schemas just as for any other data in USD.

Or does this introduce more dissonance than the dictionary-valued attribute approach, assuming that it supports actual arrays of dictionary?

20

I agree that schemas make a lot of sense for describing this data–which is the same conclusion that led to our using prims as containers in the CompoundAttributesAPI. However, we opted to not use applied schemas as they are currently unwieldy for dynamic arrays and nested attributes.

It also was conceptually fuzzy for our use case, where a lot of compound attributes are effectively separate objects in our DataCompiler description (see the slide Calvin shared from our presentation that has SMSSpriteLensFlare with a compound of SMSSpriteLensFlareInstance), as opposed to “add-ons” to a game object. The reason why these objects aren’t wholly separate prims in USD in the first place is because of the data’s heritage in Maya, where we stored all this info as compound attributes on a node.

Anway, ignoring the above case and just focusing on scalar, array, and nested struct attributes, let me finally answer your questions haha. I personally think applied schemas will always have an element of unintuitiveness compared to dictionaries since the latter actually contains the child data. But if there was an API for handling multiple-apply schemas as arrays, I think that would help. The lack of support for nested dynamic arrays would be worrisome, but in our case we might be able to simplify most classes to not be nested.