Dot Nodes in NodeGraph

Hey USD people!

A bit of a long one, so TLDR: we would like to propose encode a “dot” node in a NodeGraph; below are some options and reasonings.

We’ve been looking into the UsdShade space of USD and noticed that although there is a backdrop prim type, there is no “Dot” node prim, and no way to encode such a Prim in the node graph. These node types are common across multiple DCC’s in respect to building NodeGraphs (for shading or otherwise), and we’d like to get insight into some possible options we’re considering.

We have thought of three core approaches in mind for implementing Dot nodes with USD:

  1. A nodeShape UI Hint Metadata for NodeGraph prim which can accept dot as a valid value.
  2. A new APISchema NodeGraphDotNodeAPI which inherits from NodeGraphNodeAPI - this allows for high level schema-based filtering and further visual or behavioural customization the API, such as changing the fallback of the expansionState to closed.
  3. DotNode concrete typed schema.

By utilising the existing UIHints metadata, it would be the smallest of changes. However, “nodeShape” does not necessarily fit with all prims which want to use the UIHints schemas as they are. It could also be custom metadata; in which case it does not need to be part of the UIHints schemas, but looses some of its impact as a standard metadata between DCC’s. Also since this would be set on the NodeGraph prim, we would inherit its connection behaviour and encapsulation rules. Also with a nodeShape hint, there is further scope for allowing other shapes for a prim representation.

As always these have different advantages and disadvantages.

  1. Would be the most backwards compatible, and easy to retrofit support for, since its just another piece of metadata to read. However, it is limited in scope, and feels naïve . It also doesn’t necessarily make sense with the rest of the UIHints, since its likely only suited to NodeGraph prims, whereas the other UIHints are more globally useful.

  2. Whether its an extension of UsdUINodeGraphNodeAPI or a new API schema with inheritance, it similarly trivial to support, and even easier to add (especially if its a schema with inheritance) to existing scenes/USD builds via registration. Given it would only be applied to a NodeGraph prim, it would also function in other versions of USD, or builds without the schema, as a NodeGraph prim with its output connected to its own input; as a passthrough. We would also make use of expansion state to toggle between different view modes for the dot (either the traditional dot, or an expanded version which would look more like a “normal” node). As a new API schema it also allows for easier hiding in TreeViews, via a HasAPI query, where seeing “dot” prims is superficial. We may also extend with custom methods on the API schema to handle automatically when an attribute is connected as an input to create the matching output and connect it automatically, and vice versa.

  3. Would put the Dot node in the same space as the BackDrop and allow for more fully fledged customisable behaviour. However, I feel this is the hardest to add support for in applications prior to any official inclusion.

Personally I prefer 2, however, we are interested in the wider USD communities feedback on this one before we make an official proposal or MR. I have attached a small example material which utilises option 2.

#usda 1.0
(
    defaultPrim = "ups_textured"
)
def Material "ups_textured" (
            kind = "group"
        )
{
    asset inputs:file = @C:/Users/JamesPedlingham/Pictures/UV_Checker.jpg@
    string inputs:varname = "st"
    token outputs:surface.connect = </ups_textured/UsdPreviewSurface5.outputs:surface>

    def Shader "UsdPreviewSurface5" (
        prepend apiSchemas = ["NodeGraphNodeAPI"]
    )
    {
        token info:id = "UsdPreviewSurface"
        color3f inputs:diffuseColor = (0.36605546, 0.6923519, 0.0010682017)
        color3f inputs:diffuseColor.connect = </ups_textured/dot1.outputs:o1>
        color3f inputs:emissiveColor.connect = </ups_textured/dot8.outputs:o1>
        token outputs:surface
        uniform token ui:nodegraph:node:expansionState = "open"
        uniform float2 ui:nodegraph:node:pos = (-397, 626)
    }

    def NodeGraph "dot1"(
        prepend apiSchemas = ["NodeGraphDotNodeAPI"]
    )
    {
        color3f outputs:o1.connect = </ups_textured/dot1.inputs:i0>
        color3f inputs:i1.connect = </ups_textured/UsdUVTexture2.outputs:rgb>
        uniform float2 ui:nodegraph:node:pos = (-578, 649)
    }

    def NodeGraph "dot8"(
        prepend apiSchemas = ["NodeGraphDotNodeAPI"]
    )
    {
        color3f outputs:o1.connect = </ups_textured/dot8.inputs:i0>
        color3f inputs:i1.connect = </ups_textured/UsdUVTexture2.outputs:rgb>
        uniform token ui:nodegraph:node:expansionState = "open"
        uniform float2 ui:nodegraph:node:pos = (-578, 649)
    }

    def Shader "UsdUVTexture2"(
        prepend apiSchemas = ["NodeGraphNodeAPI"]
    )
    {
        token info:id = "UsdUVTexture"
        asset inputs:file = @C:/Users/JamesPedlingham/Pictures/UV_Checker.jpg@
        asset inputs:file.connect = </ups_textured.inputs:file>
        float2 inputs:st.connect = </ups_textured/UsdPrimvarReader_float4.outputs:result>
        float3 outputs:rgb
        uniform token ui:nodegraph:node:expansionState = "open"
        uniform float2 ui:nodegraph:node:pos = (-678, 649)
    }

    def Shader "UsdPrimvarReader_float4"(
        prepend apiSchemas = ["NodeGraphNodeAPI"]
    )
    {
        token info:id = "UsdPrimvarReader_float2"
        string inputs:varname = "st"
        string inputs:varname.connect = </ups_textured.inputs:varname>
        float2 outputs:result
        uniform token ui:nodegraph:node:expansionState = "open"
        uniform float2 ui:nodegraph:node:pos = (-988, 593)
    }
}

As well as a small image of what that may look like with “open” and “closed” states of the dot node (mock designs, example based from Katana’s existing node graph with a few manual tweaks).

Interested to hear your thoughts!

James

Hi @James – I’m interest in learning more about this. OpenUSD and DCC encodings aside, would it be reasonable to describe the dot as user interface or display metadata on the connection itself?

Hi @nvmkuruc,

I did consider storing this as metadata on a connection, however this gets complex to encode when you have multiple dots chained in a row. This is a relatively common approach especially when the user wants to make right angled straight connections across a network.

There is a possibility of encoding the connection spline points, but this would not work in an application that supports only straight connectors.

Edit:
I should also note that we do support this currently (in a non USD context), and have had requests to maintain the ability to show the dot as a node. This allows the ability to define nodes as “splice” points for later mass manipulation of shaders via downstream procedural nodes. (This is typically done via naming conventions on the node name and port).

MaterialX also has a ‘dot’ node and UsdShade should too if only because you can encode MaterialX as a UsdShade graph rather than using an external reference.

Hi @James , another good one. I’ve been trying to dredge up where we left this internally sevenish years ago when we did discuss dots in depth, internally. We came away with questions we weren’t prepared to answer yet, and there wasn’t a pressing need yet (I’m following up on what we’re currently doing in our own tooling). Some thoughts:

  1. On scope: we want not just a one-to-one Dot for graph organization, but to be able to “grab” a bunch of noodles sometimes and move them around as a unit. That ramps up the need for schema/mechanism about how the multiple connections are handled. Is this a need in your tooling?
  2. NodeGraph seems like a counter-abstraction for a Dot, as its primary purpose is to contain substructure, and a Dot very specifically must have no internal structure, or else it’s not a Dot.
  3. So IIRC, we were leaning towards a new schema in UsdShade that is not a “Container”, but also not a UsdShadeNodeDefAPI-bearing computation unit like a Shader.
  4. Why is it important that it be a UsdShade thing, given that it is for GUI/organization purposes? Because the load-bearing GetValueProducingAttributes API’s in UsdShade need to know how to “walk through” these prims without stopping. There’s also the potential there to not need to explicitly wire the outputs up to the inputs, internally, though the schema does need to stipulate the correspondence.

None of those are finalized thoughts, just initial ones.

Cheers,
–spiff

SideFX is very interested in this topic as well. Just a few thoughts to further murky the waters:

  • I strongly believe this needs to be a real prim of its own. The “splice point” use case was very important for our users when we added dots to our networks. For this to work you need to be able to connect to a dot even if the other side of the dot isn’t connected to anything, which means the dot needs an independent existence, which to me says “prim”.
  • One thing I really wish Houdini’s network dots had was allowing multiple inputs and outputs. I really think it would be wise as Spiff suggests to support multiple inputs and outputs out of the gate.
  • I would raise the possibility (or maybe this is what Spiff was getting at with his point 4) that maybe these shouldn’t be thought of as a feature of UsdShade, but a feature of Connectable Attributes. If the core Connectable APIs on UsdAttribute were aware of this special “dot” prim type that could be used to pass through (possibly multiple jumps) between an outputs and an input, that would save having to reinvent this particular wheel for UsdShade, then again for Lights and maybe OpenExec and whatever other kinds of connectable things may be added to USD in the future. Since non-GUI-manipulating callers probably don’t care about dots existing or not, at a very low level the existence of these prims could be hidden by the Connectable APIs so most callers that just care about attribute-to-attribute connections wouldn’t even need to change or be aware of the existence of dots.
  • It’s partly the above idea that makes me think that a dedicated concrete prim type (UsdConnectableDot?) makes more sense than applying an API schema to any existing prim type (be it a Shader or NodeGraph or something else).

Thanks for bringing this up James!

Mark

Thanks all for the feedback,

  1. Yes, this was a nice benefit which inheriting from NodeGraph already provided us, and was the reason for choosing to name the inputs i# and o#. I don’t believe that USD needs to concern itself with such mechanisms though, since these are typically user interaction choices. I anticipate any methods on this schema to remain simple in their concepts and to not “reshuffle” the numbers in the scenario that an input is removed, and one added later on. Since the Property ordering is stored separately, we would re-use that for visual ordering; the numbers after i or o are purely superficial to prevent duplication (A simple - last input + 1 approach could suffice). This is where having extra methods handle that behaviour would make it trivial, and if connections are manually created, that’s fine too, the next API call would create i0 as if the others didn’t exist.

  2. This makes sense to me, and leans towards option 3, a new concrete Typed schema, since API schemas cannot be assigned such rules within the connection systems.

  3. (see 2)

  4. I suspect this may have been miscommunication on my end. I didn’t intend a requirement for this to exist within UsdShade itself, but it does belong with NodeGraph (which is UsdShade). This may mean that NodeGraph could also be lifted out of UsdShade, but I feel that DotNode and NodeGraph are the two context agnostic schemas and have uses in more than UsdShade as you mention. However, saying that, I don’t anticipate using USD as a storage for such node data in the short term. As for Mark’s coments about lights, I would put these within UsdShadeNodeGraph as a concept as well. I understand they are part of UsdLux, but it should be possible to create a light from shaders (and lights are in the SdrRegistry). Either way I’m not adverse to DotNode, being outside of UsdShade, but I’d expect UsdShadeNodeGraph to be at the same “level”. OpenExec as well is an interesting one, and whether that pushes this further outside UsdShade.

  5. I would be very wary of any special schema logic that means we don’t have to explicitly connect the outputs to the inputs. This would make it very hard to have a simple way of handling multiple inputs/outputs in a robust way. This is especially important for those (like us) that need to trace the connections throughout the shading graph for certain workflows. This would also completely break any inkling of backward compatibility, and I believe that should be considered as the use of USD versions is still very wide spread.

The main thing that pushed me away from option 3 and into 2, was the backwards compatibility and ease of transition. If we make a new concrete typed schema, existing products out already will not understand these prims. The resultant shaders may still work if we follow the simple approach mentioned in 5 (retain internal out->in connections), but any internal behaviour would be lost. An API schema which inherits 90% of its behaviour and only overrides a few inherited properties (and could be codeless) is far easier to insert into older builds. I’m not beholden to this, but I believe prudent to be aware of the landscape we sit in.

Thanks again for the comments and discussion, certainly good points all around!
James

@mtucker 's note on generalization of Dot to other non-shading systems really spiked the concerns I have about the approaches we’ve been discussing, which are both practical and philosophical.

The philosophical concern is that we are forcing a UI/human-intelligibility-only concern (that also obfuscates core connectivity legibility) into the networks themselves.

The practical concern that arises from that is that we take on added processing and compilation costs arising from those UI concerns. The way we’re doing Dots internally at the studio today is clever - we have a custom OSL node that has a bunch of paired inputs and outputs that are just pass-throughs, so a Dot is just “an ordinary shader”. But in addition to the obvious that that doesn’t scale to other types of shading systems (or non-shading systems), it means we are forcing the already slow OSL compiler to have to fold those away for us - after Hydra has already had to process and encode those notes into MaterialNetworks. For OpenExec rigging, these would be costs born by the Exec compiler effectively at model-load time before a user gets to interact with the model. For sure they wouldn’t be the tallest nail, but a thousand cuts.

So I’d like to propose we turn this problem around, and recast Dot as a proper UsdUI concept that sits adjacent to a network in the same way that Backdrop nodes currently do.

Here’s the schema:

class DotNode "DotNode" (
    apiSchemas = ["NodeGraphNodeAPI"]
    inherits = </Typed>
)
{
    rel bundles
}

The idea is that, like Backdrop nodes, DotNode prims sit outside the connected network, and therefore don’t pollute the network’s connectivity at all. The bundles relationship identifies noodles that it should wrap-up and position before they proceed on to their declared destinations. The rules by which a grapher app consumes the DotNodes it finds:

  1. bundles can target any number of other objects.
  2. For every prim that it targets, it will bundle all of the prim’s property’s targets (that includes relationship targets as well as connection targets, making it useful in other domains that primarily use relationships rather than connections, as well)
  3. If a targeted prim is itself a DotNode that indicates a “chaining” of Dots - noodles will pas through the targeted prim (from the origin that specifies the connection) first, and then the DotNode that has targeted the DotNode.
  4. If a target is a particular attribute or rel, its noodles will all be bundled by the Dot.
  5. If we have an attribute or relationship that has a “fan out” (an example below from our own pipeline) and we want to bundle only some of the noodles, we have the ability (currently unused by USD OM and schemas, but supported) to target “target paths” that effectively identify particular noodles/connections of a multiply-connected attribute in a robust way.

This encoding puts a higher burden on the graphing app to display dots meaningfully, because prior to drawing any noodles, it will want to discover all the DotNodes in the container being graphed, and put them into a lookup table that we’d consult when wanting to draw any particular noodle starting at its connection-owning attribute/rel. But I argue that is exactly the right place to be paying a GUI/user-centric cost. The highly desirable properties of this encoding that we get in exchange:

  1. The network topologies themselves get to remain pure, and the dataflow simple and easy to extract and reason about.
  2. A “dangling Dot” or any other problems with your Dot data do not, therefore, compromise the network in any way.
  3. It generalizes cleanly to any type of connectable prim/schema, and does not conflate with containership schemas like NodeGraph

There are a few boundary conditions and considerations to discuss, and I’d be willing to put up a proper proposal, but wanted to float the general idea here first. Some examples:

Example 1: putting a Dot on all the connected noodles of two shader inputs on the same prim:

def DotNode "Dot"
{
    rel bundles = [</Bob_defaultShadingVariant/Looks/Material/_PbsNetworkMaterialStandIn.inputs:multiMaterialIn>,
</Bob_defaultShadingVariant/Looks/Material/_PbsNetworkMaterialStandIn.inputs:multiPresences>]
}

Example 2: Targeting only one (of two) “layer inputs” of one of our material-layering nodes:

def DotNode "Dot"
{
    rel bundles = [</Bob_defaultShadingVariant/Looks/Material/_PbsNetworkMaterialStandIn.inputs:multiMaterialIn[</Bob_defaultShadingVariant/Looks/Material/RedLayer.outputs:pbsMaterialOut]>]
}

Example 3: Chaining Dots for a more contoured shape. In this example, starting at the “owning” input multiMaterialIn, the noodles first passes through Dot1, and then Dot2 before going on to their final “layer node” destinations:

def DotNode "Dot2"
{
    rel bundles = [</Bob_defaultShadingVariant/Looks/Material/Dot1>]
}

def DotNode "Dot1"
{
    rel bundles = [</Bob_defaultShadingVariant/Looks/Material/_PbsNetworkMaterialStandIn.inputs:multiMaterialIn[</Bob_defaultShadingVariant/Looks/Material/RedLayer.outputs:pbsMaterialOut]>]
}

… and huge thanks to Doug Letterman our shading lead for the helpful discussion that resulted in these examples!

I do fear that we might be getting too complex in regards to the USD implementation to save us a little effort in the UsdImagingSceneIndex (which is already being performed in regards to NodeGraph Prims)

I had imagined that the dot nodes and their connections would be ignored when parsed into the MaterialNetwork graphs sent into Hydra. This is because they are not shaders, and do not have a shader:id (they would not use the OSL dot shader or otherwise). This is something Katana does in regards to its material network evaluation ready for rendering. The material.nodes (our material networks), omit any dot, group or switch nodes. This also would not affect the use of explicit Shader prims which do use the info:id = "dot".

This also follows the behaviour of NodeGraph prims, and can be seen already in the UsdImagingSceneIndex. If I create the network I did in my original example, but using a NodeGraph prim. The resultant shader does not require additional computation.

My fear with the bundles approach is that this will over complicate node network traversal. We can no longer follow a chain of connections to work out what is contributing or not. It requires looking at all prims at a hierarchy level just to confirm that there are no relationship bundles. We have seen networks with thousands of shaders at one hierarchy level (not nested) and that level of pre-computation to create a look up table feels overly complex compared to the alternative of ignoring such nodes when building MaterialNetworks.

Image of Hydra Network with NodeGraph, where I’d see the DotNode following a similar approach; in that it is not included in the material networks node graph.

I think maybe I did a disservice by not including the connected shaders themselves in the examples. The fear above is unfounded, as the whole point of the “bundles” design is that the Material (or other) node network is entirely self-contained. For example, regardless of how many Dots there are styling the noodle itself, Material/_PbsNetworkMaterialStandIn.inputs:multiMaterialIn is directly connected to Material/RedLayer.outputs:pbsMaterialOut, and the UsdShade API’s, Hydra ingest, and any network editing or JIT splicing get from consumer node to producer node in a single hop.

Dealing with networks of thousands of nodes in Flow and Katana (and Hydra) is our primary concern, motivating the “bundles” approach. It’s true that going with a NodeGraph-like node in the network would be boiled out at Hydra-ingest time, but evaluating connections in USD is not cheap (we do plan to work on that), and it obfuscates the network’s true connectivity, which will be felt not just by ingest code like Hydra, but anyone tracing connections in usdview through the context-menu, or just looking at the scene description.

NodeGraphs in their existing form serve an organizational purpose that goes beyond screen presentation, as they’re the mechanism of packaging and reuse, and hopefully in the future can also carry NodeDefAPI-conforming implementations. So it feels more legitimate that Hydra takes an extra hop through each one of its interface points to find the attributes/nodes that are producing values. But Dots are of no more use or concern to rendering or network splicing than Backdrops, and thus the argument that Dots should play no role in network decoding at all.

The pre-processing of Dots that is needed only to draw styled graphs is absolutely alot more complicated than what UsdShade graphing algorithms need do today. We can provide utilities that will build up those tables. But the thing I want to emphasize is that keeping Dots out of the network protects the simplicity and scalability of Materials that will be processed by their hundreds or thousands at once in a render; the process/logic that gets more complicated (and likely a little slower) is concerned with the one or small number of Material networks that a user can profitably work with at any one sitting, so it doesn’t need to scale.

Our perspective is that the rendering/JIT-editing/data-integrity needs outweigh the extra complexity on the UI-tooling side. But if we’re all on the same page about what the tradeoffs are, that’s the discussion to continue!

I’ll note also that encoding Dots as metadata would be another way to achieve the same goals we’re valuing (because that also wouldn’t obscure network topology). We haven’t been able to think of a robust way to do it with the OM we currently have, but if anyone has ideas in the space that might include “if we could just do X” that could be worth discussing, also?

Fwiw I do like Spiffs suggestion of encoding the dots independent of the shading graph.

Not having to add extra things to parse in graphs for shader construction at runtime is a win.

I am fond of the idea of solving this by exposing this in a reasonable place in the OpenUSD repository. Spiff’s suggestion of introducing it in In UsdUI sounds like a reasonable place, so systems that do take advantage of it during none runtime moments, such as the UI can have it exposed to them easily.

Lookup table sounds reasonable and doing the discovery early on, but I wonder how easy it will be to map into / resolve the lookup table as expected or at least make it easy to parse / pull together appropriate connections during network construction. If it is as easy as Spiff suggests.

Based on what I have seen, then JIT benefits far outweigh the cost of needing to include it in runtime loading / execution that does not need it.

In general, after going through the discussion I think it would be great to include this.

I’m confident that we can efficiently provide something that answers the query “Given a connection from destination attribute /a/b/c.dest to attribute /a/b/f.src, what is the chain of Dot nodes that noodle should pass through?”. How to represent the Dot nodes in a way that is meaningful to a prospective graphing application that is already processing “pure” UsdShade networks is less clear, and would likely benefit from someone doing some prototyping. One possibility would be @pablode 's usdshadeview plugin ?

I built a system that sounds incredibly similar to Spiffs suggestion here in the Material system I build at Imageworks, and I took that design decision for very similar reasons, wanting the material network topology to remain “pure” and to keep the processing efficient. The concept is pretty reasonable and fairly straight forward to implement… until you get to composition, we didn’t really have full USD composition in our system, just something a lot closer to Katana attribute inheritance, and editing at multiple locations in the nodegraph, but just that simplified “composition” made the managing of the “sidecar” dot nodes a lot more fragile. It was really easy for folks to break the data model. I don’t think the USD representation should under-estimate the app/UI complexity involved in robustly tracking this side-car data.

I think that Dots (more so than Backdrops) should be considered a first class component of the material graph - I’ve seen many graphs where without the Dots the graph becomes unintelligible to the human, and some where the artist have spent nearly as much time laying out the nodes and connections (with Dots) as they have building the material - to ensure its maintainable by their team.

We may want to decorate the Dot nodes with some metadata that allows downstream tools to locate them in the material and “do work” - perhaps they are used as texture-baking bake points or some other material optimization system that more complex than a variant could provide. Granted these cases could be handled with a “Constant” shader node with some additional metadata too - but there are some advantages to having a different semantic for these - they can have “no value” when no upstream connection exists.

I’m not completely opposed to the side-car data approach - we made it more or less work at Imageworks - but it was hard, with a much simpler composition system in play. I would really want to see a complete examination of the corner-cases before we committed to this direction.

This is where I have a misunderstanding. From our users we have seen them use the dot node (and its connection points) as the point in which to splice in shaders; much like a NULL node in other applications. They can name the dot to fit a certain convention and it acts as a neutral point at which any arbitrary shaders can be spliced without necessarily having to know the exact names of the shaders inputs and outputs.

Additionally dot nodes being part of the network allow splicing into points that are not yet fully connected. A user may put a dot node as an input to a shader, but not connect it yet, understanding that there may (or may not) be further shaders spliced in later on.

As mentioned I am likely misunderstanding something, so I’m all ears (or eyes in this case).

I am also not certain that the calculations of material networks to ignore the dots is a pain point in rendering or evaluation today. Katana has and continues to do this with its networks with known shader counts in the high thousands with likely similar numbers of dot nodes. We must already parse the full network, skipping certain prims is near zero cost, as is already done with NodeGraph prims. What we are concerned about is the artist interactivity of the UI session which we have seen become a pain point when needing to evaluate and draw shallow networks with thousands of shaders. We must ensure that the artist experience of loading the network, and editing it, is as interactive as possible. The caching you mention may help there, but will it also increase composition time if that is where it occurs? If a connection changes in the network, must we re-cache everything each time?

We already cannot rely on the Value resolution of USD for shader properties, since the values are not discernible without running the shader itself. Thus I don’t think these would be “evaluated” as such. Perhaps this pushes two different approaches, one for UsdShade, and one for general non UsdShade NodeGraphs.

If we do not want DotNode’s to be their own Prims (or are OK with that limitation), then another approach could be similar to Matthew Kuruc’s suggestion. We could encode the dots as a “spline” on the connection as metadata. This would be able to provide another benefit of not inflating Prim count itself; and similarly avoid inflating the true connections. Multiple dots could be achieved as multiple spline points. This would likely match whiteboard like tools where the connections themselves can shaped independently.
This would again loose the benefit of being able to splice into and be able to connect to the dot rather than the actual shader.