It would be interesting to know if it is reasonable to simply “remove” the data from the assetPath or if I should handle it differently or even take it into account.
Any input would be interesting because most of the time, there is a reason that data is given to the resolver, so I assume there’s something I should do with it here.
Manny, thanks for reading.
If you read it on creation day, have a great weekend. If not, just have a great Day
Hi @Root42 , no ArResolver should need to worry about SDF_FORMAT_ARGS . In the normal course of operation of all shipping OpenUsd asset-processing code, which includes stage composition/population, and UsdUtils utilities for dependency analysis and asset localization and packaging, the client of Ar strips the Sdf arguments from the string before handing it off to the ArResolver to resolve.
Any script or code that wants to perform “raw” processing of a layer’s assetPaths using Sdf API’s is on the hook to do so as well. It feels like we should document this strongly somewhere, though I’m unsure exactly where - open to suggestions!
i guess the first place i would look is DevGuide and then Spec (Maybe a section that describes how loading and writing works in a general ?) i assume the later is intended to be a bit like the c++ ISO standard description what everything should be and how it should work. (im guessing because that not what its currently doing) and then i would maybe even link it into AR just for good measure.
but take it with a grain of salt as i encountered it the first time while working with @BigRoyNL sooo maybe other people would look elsewhere.
also you mentioned that it should not reach the resolver if i understand you right
no ArResolver should need to worry about SDF_FORMAT_ARGS
but when i run the resolver within SideFs Houdini 20.0.710 using a tool developed by @BigRoyNL and/or the reference node (its a bit of a cheap test i know) then im greeted with the following
ps: getAsset() is a custom function that caches server communication just to avoid confusion.
ps: i just did a test using “Raw” usd23.05 and got this (the test is just a stage that loads a payload and the payload then has the uri in it, same as in Houdini)
if i understand you right the SDF_FORMAT_ARGS should not reach the _Resolve() method or did you mean something else by ArResolver should need to worry and im just miss interpreting ?
To make sure Houdini isn’t doing anything wrong here, could I get more information about this tool you’re referring to?
Houdini uses :SDF_FORMAT_ARGS: extensively when importing data from SOPs and BGEO files into USD, and I haven’t heard any reports of these args making it into resolvers, so I’m hoping that this is a bug in this “tool” you’re referring to…
hello mtucker.
the upper example is both the normal reference node the “tool” is just a wrapper around the same node (just pices together the Uri)
also: the second example the one with the exact same output is just “raw” usd
im gonna append an Zip with the specific file used for the example. this will also have the python script used for Raw usd testing (the path in it is a “fake” path i used an absolute path but in order to put it all into the same zip i did it as a relative path not sure if the relative path is correct tho),
i can ship the resolver if you want but as it needs the AYON-Environment it will probably not work for you.
if you have a resolver that has the print outs in the functions would be cool if you could just see what happens for you.
Hello there again.
i realized i forgot to append the zip in the other message so im gonna try to build a very small reproducible example here.
Testing notes
so my test is constructed from 3 parts.
A python script that just runs Usd.Stage.Open on the test_entry.usd file.
the test_entry.usd file is just the file that uses the payload.usd
and the payload.usd finnaly then sublayers the “paths” or Uris
here are some things i tested inside the payload.usd:
test.usd:SDF_FORMAT_ARGS:order=100 - dosnt strips off SDF_FORMAT_ARGS
./test.usd:SDF_FORMAT_ARGS:order=100 - strips off SDF_FORMAT_ARGS
/path/to/test.usd:SDF_FORMAT_ARGS:order=100 - strips off SDF_FORMAT_ARGS
C:/path/to/test.usd:SDF_FORMAT_ARGS:order=100 - dosnt strips off SDF_FORMAT_ARGS (under Linux did not test with windows, i assume under windows this is an absolute path)
ayon://path/to/test.usd:SDF_FORMAT_ARGS:order=100 - dosnt strips off SDF_FORMAT_ARGS
every time a “wrong” relative path is created. The SDF_FORMAT_ARGS are not removed eg ./file.usd will have them removed but file.usd will not have them removed.
im not sure if that’s wanted or not.
i also want to mention that the example paths i showed atop dont work if i just put them into Usd.Stage.Open(), (i belie i got it to not remove the ARGS once but i have no record of this and cant reproduce it so i will chuck it up to probably just ran the wrong test)
also i just want to mention. the resolver we have is not registered as a URI resolver normally. i however did a test where i ran the test with the resolver resisted with "uriSchemes": ["ayon"] enabled and came to the same output
thanks for reading and especially your time with this issue.
Sorry for the long delay. But having dug into this a bit, I am also having trouble squaring what @spiff says with what I’m seeing when debugging the code…
I can replicate the fact that Resolve is called with format args when the path is test.usd:SDF_FORMAT_ARGS:order=100, but not when the path is ./test.usd:SDF_FORMAT_ARGS:order=100. Again, this is happening during composition and seems to be a result of how ArDefaultResolver treats search paths as a special case, calling Resolve on the asset paths in cases where relative paths do not call Resolve.
When using full URL paths (I used opdef://... instead of ayon://... because that URL is handled by Houdini’s asset resolver, but I assume ayon: URLs would be handled by the USD library similarly) I again got format args sent to calls to CreateIdentifier, but never in calls to Resolve. If the ayon resolver is getting Resolve called with format args, that is the one situation that does not match what I’m seeing.
As a side note, @Root42, I think you should definitely register this as a URL resolver. You cannot assume that as a non-URL resolver that this plugin will be used as the “global” resolver for all non-URL paths. And if it is used as that, you run the risk of breaking the user’s pipeline if they have their own “default” resolver.
All this to say that I think I have to pass this back to Spiff for some clarification I don’t see any evidence that Houdini is the source of any of the oddness here.
Thanks for verifying the behavior in Houdini, @mtucker ! It turns out that what I said above is what we expect to be happening, but in fact (to my surprise) Sdf itself is not actually performing the identifier splitting prior to calling Resolve; it expects its clients to do so, without explicitly stating that, and it sounds like there’s at least one call site in Pcp that is not.
Regardless, we believe Sdf should be doing this itself, and we’ll take care of that for the next release.
First of all, thanks for testing and your in-depth response.
I can replicate the fact that Resolve is called with format args
Okay, good to see that you can replicate it (well, not good, but good that we know more now)
I again got format args sent to calls to CreateIdentifier, but never in calls to Resolve.
Okay, I checked this again, and yes, indeed. The call to resolve is fine. But I call the internal _Resolve() method from the CreateIdentifier() method so that I can create the identifier from the resolved path so that USD compares based on the resolved path. Then, I handle the call the “Raw” way. Sorry for missing this in my test.
As a side note, @Root42, I think you should definitely register this as a URL resolver.
I will discuss that with the team to see how we go. I think the point you make is really good. And maybe I will just have a variable in the build setup that allows you to register it as URL or as Global and then remove unneeded code via #ifdef.
Regardless, we believe Sdf should be doing this itself, and we’ll take care of that for the next release.
Hi @spiff, thanks for the info about this. I guess it is time to expand the test cases ;-).
Just out of interest, will this end up as a PXR internal issue, or will there be a GH ticket? I’m just interested in seeing what the actual changes might be and if there is a discussion. It’s not important, but it’s always fun to see.
Both of you have a great week. and many thanks for your time. Cheers Lyon
Hi @Root42 , you’re definitely slightly voiding the warranty in trying to resolve inside CreateIdentifier SdfLayer must include the FORMAT_ARGS in the string it passes to CreateIdentifier, because two otherwise equal asset paths may produce very different layer contents based on FORMAT_ARGS. So if you are stripping them yourself in order to resolve, I’d suggest you put them back on when you’re done?
I did file an internal issue for the SdfLayer problem you encountered, and we should get to it soon - thanks!
Thanks for the info. That’s a good point.
I just checked, and when an AyonUri is found, we return the URI. I’m simply not going to remove the SDF_Format_Args from it. (PS: I was resolving the anchoredAssetPath like the default resolver dose just recently switched around some stuff)
@sunya reminded me to ask why you needed to do your own resolver-based path deduplication, because the internal SdfLayer registry already does use the resolved path (plus format args) to deduplicate layers.
We don’t do it for deduplication, per se. We encountered that Houdini (mainly) is very dynamic, thereby causing re-resolution when reconstructing its scene.
Setting up the resolver with a cache object helped quite a bit.
The cache object is also shared, so we don’t have to care if something creates a new resolver because even a new resolver will pick up the “Global” cache while still being able to opt out of doing so
If I remember correctly, Luca Sheller went the other way around, where every resolver has its cache, and he used a Global std::map to avoid creating unneeded Resolvers.
There is also a second advantage: we are currently working on pinning support for farms to write out a file that gets loaded at startup to avoid communication with the server. Having a separate cache for all (all resolvers that don’t opt-out) makes this relatively simple to implement.
Oh, and before I forget, there is even a third advantage: we aren’t finished with the development of the feature on the Usd side, but our server allows us to resolve large amounts of Uris in one batch, which is a lot more performant than doing them one by one. We want to have a system in place that simply has a “file” or “Db-Info” that knows all the things related to the scene so that we can do one big server request instead of many small ones at once.
There are more ways to do the same job, but this one was simple and worked well, so I opted for it.
Ps: there is also the idea of doing “redirection” for nested version updates via the resolver, and having our “own” data structure that we can easily modify is quite handy for that, too. (I am not sure if you remember this thread: Updating nested asset paths - recommended workflow)
Sorry for the lengthy answer; I didn’t get it smaller. I hope this all makes sense and sounds reasonable.
I don’t see any evidence that Houdini is the source of any of the oddness here.