USD Asset Resolver - Practical Example/Reference Implementations

Community USD
,
1

UsdAssetResolver_Logo_wShadow_wTitle_2048_1024
UsdAssetResolver_Logo_wShadow_wTitle_2048_10242048×1024 96.4 KB

Hi everyone,

as part of the USD Survival Guide (Forum Post / Website) there is an asset resolver sub-section. Apart from giving an overview of what the asset resolver is/does, it also has some example resolvers.

In the last few weeks I’ve added some more updates and made the install process for Houdini easier, so I’d thought I’d re-share it here (separately from the guide post), to make it more visible in the forum.

You can now “1-Click” install the example resolvers via the update manager:
UsdAsserResolverUpdateManager

Here is the documentation on how to run it: USD Asset Resolver - Automatic Install

I hope this is useful for smaller studios in helping them get up and running with USD without having to do C++ coding.

Currently the repo holds these resolvers:

  • Production Resolvers
    • File Resolver - A file system based resolver similar to the default resolver with support for custom mapping pairs as well as at runtime modification and refreshing.
    • Cached Resolver - A resolver that first consults an internal resolver context dependent cache to resolve asset paths. If the asset path is not found in the cache, it will redirect the request to Python and cache the result. This is ideal for smaller studios, as this preserves the speed of C++ with the flexibility of Python.
  • RnD Resolvers
    • Python Resolver - Python based implementation of the file resolver. The goal of this resolver is to enable easier RnD by running all resolver and resolver context related methods in Python. It can be used to quickly inspect resolve calls and to setup prototypes of resolvers that can then later be re-written in C++ as it is easier to code database interactions in Python for initial research.
  • Proof Of Concept Resolvers
    • Http Resolver - A proof of concept http resolver. This is kindly provided and maintained by @charlesfleche in the arHttp: Offloads USD asset resolution to an HTTP server repository. For documentation, feature suggestions and bug reports, please file a ticket there. This repo handles the auto-compilation against DCCs and exposing to the automatic installation update manager UI.

Here are some resource links:
Website: USD Asset Resolver - GitHub Pages
GitHub: GitHub Repository
USDSurvivalGuide: Asset Resolver - Section

Currently we only auto-compile against Houdini, if someone from the other major DCC vendors is interested in adding their app to the list, please get in touch :slight_smile:

A video tutorial showing how to install and customize some of the resolvers, will follow some time in the next week. (Everything is also already covered in written form in the docs already.)

I’d love to hear your feedback! If you stumble upon any other interesting open source resolvers, let me know and we can try adding them too. I can also give a quick live demo in a working group if anyone is interested.

Cheers,
Luca

16 Likes
2

This is a great addition to an excellent resource. Thanks!

1 Like
3

Hi everyone,
there is now also a video tutorial available on how to install (and optionally build) the OpenUSD asset resolver production/reference implementations that are available in the above mentioned Usd Asset Resolver repository.

The tutorial covers how to use the automatic installation manager for Houdini, to easily install all resolvers with a few clicks, as well as how to build them yourself. We also look at how you can customize them to fit your needs without having to code in C++. This is especially useful for smaller studios, who often don’t have the C++ development resources.

You can view it here:

You can also view tutorials on our tutorials section in the UsdAssetResolver documentation.

Let me know what you think!
Cheers,
Luca

4 Likes
4

Thank you so much for this Luca

i successfully installed and test the resolver, and all test and examples works as expected, i’m trying to figure out how this can be used in our pipeline, but not sure how to “attack” this.

I got a path that looks like this
@./asset/char/mr-X/model/pub/latest/mr-X-model-latest.usd@

we want this path to resolve to an actual version on disk, so something like this.
F:/myMovie/char/asset/mr-X/model/pub/v004/mr-X-model-v004.usd

So in the case above we want to replace the word latest with v004 , and then add the achor path dependent on the os .

i guess we could use the mappingPairs and then add one entry for each layer in my stage, but imagine this could be a pain to maintain when new versions are published, and new layers are added to my stage. The mappingPairs has to be dynamically updated whenever something changes.

We are probably going about this in the wrong way, so any suggestions on how latest can resolve to an actual version on disk is appreciated

Btw. we talked at siggraph, thank you again for the your greate talk there :smiley:

5

Ok i think i understand this better now, the mappingPairs could probably be a good way to pin versions, and maybe this is also the intension behind it?

For resolving latest version, there’s probably better ways of doing this i guess. I played around with the Python Resolver, and got it to do what i want.
But as you stated, it should probably not be used in in bigger production scenes.

I also looked into the Cached Resolver, it almost does what i want, but i need access to the full assetPath. In other words, say i got a layer that is relative to another layer, like in the image bellow. Is there a way i can get the resolved full path inside ResolveAndCache?

image
image867×197 20.3 KB

I need access to the full path so i can extract the name of the asset and the asset type from the full path.
assetType = char
assetName = mychar

Thanks

6

Hey, sorry for the late reply.

So the idea is for non-pinned entries to be resolved and cached via the
Python exposed ResolveAndCache method.

The reason why it is currently not working is because the “CachedResolver” doesn’t pass relative paths (anything starting with “./” or “…/” through the ResolveAndCache), only paths that don’t start with “/”, “./” or “…/”. This is intentional to allow fast C++ based resolving of relative paths. (See the features here for more info.)

Someone else also contacted me recently and asked if we can add relative identifier support too, I’m currently working on it, should release some time in the next week. I’ll add some more production relevant examples too to make it easier to understand.

So your solutions are:

  1. Make all identifiers that should go through Python not start with “/”, “./” or “…/”
  2. Wait for the above mentioned update

Going with 1. is actually cleaner, because for 2. to work, we have to pre-process the identifier since the USD resolver resolve method is not anchor path aware. That means the update will expose hooking into the CreateIdentifier method and not the Resolve method. I’ll add more docs on that with the update.

Ok i think i understand this better now, the mappingPairs could probably be a good way to pin versions, and maybe this is also the intension behind it?

Yes that should be used for pinning.

About the OS based anchor path, that is something we could add in C++ too, I’ll keep it in mind for future updates. For now I’d recommend making your asset path this @asset/char/mr-X/model/pub/latest/mr-X-model-latest.usd@ and adding the OS drive ext in the ResolveAndCache method (or when you add a mapping pair value).

Sorry if this answer seems a bit chaotic, it should become clearer once the update is released.

Cheers,
Luca

1 Like
7

Hey,
I’ve just released the update that now allows exposing relative file paths to Python on the CachedResolver.

All you need to do is set the “AR_CACHEDRESOLVER_ENV_EXPOSE_RELATIVE_PATH_IDENTIFIERS” environment variable to “1” or “true”. Then the relative identifier creation is also run through Python and cached.

I’ve also added a small production case study example to further explain it and how it can be used as a opt-in mechanism to override relative paths.

Let me know if that solves your issues and if the documentation is understandable. If not I’m happy to enhance it :slight_smile:

Links:
CachedResolver - Docs - Relative File Path Identifier
CachedResolver - Production Case Study

2 Likes
8

Thank you so much Luca, this works perfect :slight_smile:

I was able to compile the resolvers for Maya 2024 on windows, i can share the modified cmake files with you.

Maya 2022 on the other hand, did not work, i think the reason may be that its using a older usd version 21.11. I will look closer into this as we are currently on maya 2022.5

Thanks Again

2 Likes
9

Nice, feel free to make a PR in the repo :slight_smile:

Mh yes that may require the AR 1.0 spec. I don’t plan on backporting the resolvers to support that any time soon though, sorry :confused:

10

CMakeLists.txt (10.5 KB)
Hi Raymond,

I’m attempting to compile the resolvers from: VFX-UsdAssetResolver@htps://lucascheller.github.io/VFX-UsdAssetResolver/
as well.
But I’m running into some errors.

Can I ask you a few details of the CMakefile modifications?

What modifications and packages/DCC/git clones did you use and specified in CMakeList.txt?

Here’s a summary the ones I’ve made and the output log results of the make:

Using VS2019 (vs142)
Maya 2024.2
OpenUSD 0.22.11/USD
Also tried with download prebuilt v0.26.0 from :htps://github.com/Autodesk/maya-usd/tree/release/v0.26.0

CMakefile changes:

PXR USD include/libs (AR_PYTHON):

git clone and build version from OpenUSD v0.22.11:
compiled with VS2019 to my D:\build_openUSD_vs2019

lib: D:\build_openUSD_vs2019\lib
includes :D:\build_openUSD_vs2019\include
I’ve also try using the downloaded Maya USD version 0.26.0 from: htps://github.com/Autodesk/maya-usd/tree/release/v0.26.0 and
CHANGED: AR_PXR_LIB_PREFIX to “usd_” (From “pxr_” or “libpxr_”)

Python srcs (AR_PYTHON):

Python lib/includes, using Maya2024.2’s python 3.10 included bins and includes:
Python lib: C:\Program Files\Autodesk\Maya2024\lib\python310.lib
sitepackages:C:\Program Files\Autodesk\Maya2024\Python\Lib\site-packages

Boost srcs (AR_BOOST):

Boost src and includes:
Boost: from downloaded Maya USD build v0.26.0 from @ htps://github.com/Autodesk/maya-usd/tree/release/v0.26.0 and
installed to: C:/Program Files/Autodesk/MayaUSD/Maya2024/0.26.0_202311130904-e634cda/

change AR_BOOST_NAMESPACE to boost (From hboost)
lib name:boost_python310-vc142-mt-x64-1_76
lib path: C:/Program Files/Autodesk/MayaUSD/Maya2024/0.26.0_202311130904-e634cda/mayausd/USD/lib
includes: C:/Program Files/Autodesk/MayaUSD/Maya2024/0.26.0_202311130904-e634cda/mayausd/USD/include/boost-1_76":

As mentioned before,For PXR USD and BOOST usd includes, I’ve optionally try using:

  1. cmake’d pixar USD git download (OpenUSD) : my D:\build_openUSD_vs2019\includes
  2. Downloaded prebuild and installed Maya USD v0.26.0 from htps://github.com/PixarAnimationStudios/OpenUSD. ie: C:\Program Files\Autodesk\MayaUSD\Maya2024\0.26.0_202311130904-e634cda\mayausd\USD\devkit.zip’s version.

The output log (and errors) I’m encountering are:

C:\developer\testdev\VFX-UsdAssetResolver>cmake --build build --clean-first --config Release
Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

Microsoft (R) Build Engine version 16.11.2+f32259642 for .NET Framework
Copyright (C) Microsoft Corporation. All rights reserved.

1>Checking Build System
Building Custom Rule C:/developer/testdev/VFX-UsdAssetResolver/src/FileResolver/CMakeLists.txt
cl : command line warning D9025: overriding ‘/W1’ with ‘/w’ [C:\developer\testdev\VFX-UsdAssetResolver\build\src\FileRe
solver\fileResolver.vcxproj]
debugCodes.cpp
resolver.cpp
resolverContext.cpp
D:\build_dec10_2019\include\pxr/base/gf/ilmbase_halfLimits.h(66,17): error C2059: syntax error: ‘)’ [C:\developer\testd
ev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/ilmbase_halfLimits.h(66,1): error C2334: unexpected token(s) preceding ‘:’; ski
pping apparent function body [C:\developer\testdev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/ilmbase_halfLimits.h(109,1): error C2143: syntax error: missing ‘)’ before ‘;’
[C:\developer\testdev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/ilmbase_halfLimits.h(109,1): error C2059: syntax error: ‘)’ [C:\developer\testd
ev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/ilmbase_halfLimits.h(109,1): error C2238: unexpected token(s) preceding ‘;’ [C:
\developer\testdev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/traits.h(31,1): error C2059: syntax error: ‘namespace’ [C:\developer\testdev\VF
X-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]

              <..  truncated middle log section...>

D:\build_dec10_2019\include\pxr/base/gf/vec2h.h(65): message : see declaration of ‘std::pxrInternal_v0_22__pxrReserved_
::GfVec2h’ [C:\developer\testdev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/vec2h.h(304,1): error C4430: missing type specifier - int assumed. Note: C++ do
es not support default-int [C:\developer\testdev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/vec2h.h(305,5): error C2065: 'data’: undeclared identifier [C:\developer\testd
ev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/vec2h.h(305,23): error C2676: binary ‘[’: 'const std::pxrInternal_v0_22__pxrRes
erved::GfVec2f’ does not define this operator or a conversion to a type acceptable to the predefined operator [C:\dev
eloper\testdev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
D:\build_dec10_2019\include\pxr/base/gf/vec2h.h(305,23): fatal error C1003: error count exceeds 100; stopping compilati
on [C:\developer\testdev\VFX-UsdAssetResolver\build\src\FileResolver\fileResolver.vcxproj]
resolverTokens.cpp
Generating Code…

C:\developer\testdev\VFX-UsdAssetResolver>cmake --install build
– Install configuration: “Release”
– Installing: C:/developer/testdev/VFX-UsdAssetResolver/dist/fileResolver/resources/plugInfo.json
CMake Error at build/src/FileResolver/cmake_install.cmake:55 (file):
file INSTALL cannot find
“C:/developer/testdev/VFX-UsdAssetResolver/build/src/FileResolver/Release/fileResolver.dll”:
File exists.
Call Stack (most recent call first):
build/cmake_install.cmake:37 (include)

C:\developer\testdev\VFX-UsdAssetResolver>
############ END OF LOG ###########

< END OF LOG Segment >

11

Hi Markus,

I made a clone with my tweaks to make maya 2024 work.

In the build.bat file adjust the paths to point to where the mayaUsd devkit is located

hope that helps :slight_smile:

12

Hey, thanks for Maya implementation!

Just for my understanding: Do the releases in the GitHub - Autodesk/maya-usd: A common USD (Universal Scene Description) plugin for Autodesk Maya repo contain all the needed libs and headers (as far as I can see yes?).

If yes, I’d love to create an auomatic build workflow against Maya similar to Houdini (with the changes from your repo @Raymond).

(If anyone from Autodesk reads this: Is there an API do download Maya (or Autodesk products), so that they can be used for automatic build workflows?)

Cheers,
Luca

EDIT: While the devkit/Usd dependencies are there, we still need the Maya’s Python installation. So looks like this is not possible to add (unless I can somehow download Maya withouth needing a license. If anyone has some pointers if this is technically and legally possible, I’d be happy to finish the implementation, keeping it on a branch for now)

1 Like
13

Hi!,there is a guy for dummies in windows?all i want its to update assets to latest version on a disk,something like wildcards could be cool
Thanks!

14

Hi Ruso65,

It depends what you are going for and how your file structure is.

It should be easy to add: Just add a file glob in the CachedResolver Python file that searches for the latest version.

Not sure exactly what you mean with wildcards, could you define exactly what you are going for?

Cheers,
Luca

15

Hi Luca! First of all thanks for your guides and reference Resolver implementations - they’ve been a great resource so far.

I’ve been playing with the cachedResolver, figuring out how we may use it or something like it in our medium-sized studio, and have run into a curious issue specifically in Houdini.

I am testing doing a bulk database query in the Initialize method to create CachedPairs for the latest of each asset.

The initialization works, and if I manually run consecutive resolver.Resolve("<asset>") commands I get the expected behaviour. However if I drop a reference node in Solaris and set the path to some identifier, Initialize runs again, and runs every time I edit the identifier, even if setting it to a previously cached value.

I’ve confirmed this with logging and checked the UnitTestHelper.context_initialize_call_counter and it is ticking up with each Resolve.

Do you (or anyone else) know of any reason why this would happen when using the Resolver via a Solaris node, but not through pxr.Ar?

16

Hey, thanks for checking out the resolver.

Thanks for the bug report, it is due to a bug of how I store shared contexts, I’ll release a fix in the coming days. This should fix the double init call.

The strange thing is, I still get a repeated _CreateDefaultContextForAsset call for everytime I duplicate a LOP reference node. This is what I have done for debugging so far:

If you set the following env var:

export TF_DEBUG=CACHEDRESOLVER_RESOLVER_CONTEXT

You’ll see it being run

Resolver::_CreateDefaultContextForAsset(‘/Some/pinning/file/path.usd’)

twice. The resolver has a safeguard against this to re-use the same context (VFX-UsdAssetResolver/src/CachedResolver/resolver.cpp at 4ae6d4eb48541560f19515951a2db9c742ae9fc0 · LucaScheller/VFX-UsdAssetResolver · GitHub), the log will show that via:

Resolver::_CreateDefaultContextForAsset(‘/mnt/data/PROJECT/VFX-USD/VFX-UsdAssetResolver/files/generic/mapping.usda’) - Reusing context on different stage

I am guessing we get multiple calls, due to how Houdini handles stages (per node/viewport etc.).

So, so far, so good (I never got around to filing an RFE to SideFX to automatically re-use the context, this shouldn’t be something the resolver has to handle in my opinion. Feel free to file RFE for this, if you have the time).

Now to the problem:

If we create the reference via the reference LOP node, it does indeed call _CreateDefaultContextForAsset on the reference itself. I am not sure why this is happening. If you create the reference via:

from pxr import Sdf
prim = stage.DefinePrim("/test")
prim.GetReferences().AddReference(Sdf.Reference("/path/to/file.usd"))

It doesn’t, this is what I would expect as the default behaviour. So the reference node seems to be doing something unexpected?

Cheers,
Luca

1 Like
17

@JoeBrown Actually found some time today to fix it.

The new release is deployed here:

(Or installable via the standard Update Manager). Can you check and see if it fixes you issue?

If you have the time, could you still file an RFE/Question to SideFX to ask why the _CreateDefaultContextForAsset is called on the files loaded by the reference LOP and let us know the answer here?

18

Thanks for the super quick response. I updated our build from 0.6.1 to 0.7.0 but I’m still getting it re-initializing every time.

One of our team described the problem to SideFX on their slack and they replied:

I don’t think we’ve seen this reported before. If you’re observing strange behavior like that with the Reference LOP, I’d double-check the performance monitor, add the Cook Count column, and double check that something (an expression in a parameter or some callback) that a user or tool has added, isn’t re-cooking a node unnecessarily, multiple times.But if you are seeing funkiness please log a bug with any logs or example files

So we’ll do the checks they describe and to rule that out, and let you know the results.

19

I started to do some profiling, having removed all our custom tooling besides the asset resolver and the core db library needed to query for resolutions. I have some confusing observations.

For the sake of experimentation, I am caching all valid identifier:path pairs (some 15,000) in the context during the ResolverContext.Initialize function - this takes about 8 seconds when using pxr.Ar directly. In this test I am never hitting the ResolveAndCache function.

First observation is that I’m seeing ResolverContext.Initialize run twice every time I create/edit the identifier on a LOPs Reference node, but only if I enter a new identifier. If I enter a previously entered identifier (on the same or different Reference node), the node cooks but the Context doesn’t re-initialize.

The node cooks just once per edit.

I can run the following and Resolve as many identifiers as I want and never hit Initialize after the first time:

from pxr import Ar
ar = Ar.GetResolver()
ar.Resolve(identifier)
20

I know you are debugging, but I just wanted to call out for visibility that the ArResolverScopedCache was designed for caching Resolve() calls, to give clients finer (and at multiple granularities, since they scope/stack) control over when caches need to be reevaluated.

Following this thread with interest!