Hello,
I’m trying to find a way to reset USD memory cache between tests so each test gets a clean slate. currently, if two tests try to make a stage with the same name then one of them fails.
from pxr import Usd, UsdGeom, UsdShade, Sdf, UsdUtils
usd_path = "c:/temp/test123213.usd"
#one test can create a temp stage:
stage = Usd.Stage.CreateNew(usd_path)
#then another one may create the same one and fail
stage = Usd.Stage.CreateNew(usd_path)
I tried:
UsdUtils.StageCache.Get().Clear()
but it didn’t fix it.
the problem gets very annoying the moment there is a shared fixture that generates a new stage as it has to be set to session level otherwise it fails.
Error in 'pxrInternal_v0_22__pxrReserved__::SdfLayer::_CreateNew' at line 494 in file C:\b\w\ca6c508eae419cf8\USD\pxr\usd\sdf\layer.cpp : 'A layer already exists with identifier 'c:/temp/test123213.usd''
some obvious hacks are to generate unique names for each test or use session-level fixtures. However, this doesn’t give me any confidence in the tests as I have no idea what else might be shared between them. Ideally, I would like to call reser/clean to start fresh in each test.
This is due to the fact that the first handle is still in memory in a way.
from pxr import Usd
usd_path = "c:/temp/test123213.usd"
stage = Usd.Stage.CreateNew(usd_path)
del stage
stage = Usd.Stage.CreateNew(usd_path)
This works for me. It seems similarly done in USD tests.
Note that in Python the variable stage is still the old instance, because Python first evaluates the expression on the right hand side Usd.Stage.CreateNew(usd_path) and then goes to assigning it as new value to stage. Up to that point, the old stage is still in memory. It seems just assigning stage = None in-between didn’t fix it, but it’s good to know that your example code may have suffered from that.
I initially thought that this may have worked as well:
Alternatively, you could use stage.CreateInMemory() as many times as you like, and either Export() or GetRootLayer().Export() if/when it is required to serialize the root layer contents (or the flattened stage, in the first case) to filesystem/storage.
My example was heavily simplified. The CreateNew might be called in totally different files in unrelated tests.
if I had access to stage variable I could do this: UsdUtils.StageCache.Get().Erase(stage)
to remove it from the cache. My problem is that I have access to it.
Let’s imagine a scenario like this:
def test1():
usd_path = "c:/temp/test123213.usd"
stage = Usd.Stage.CreateNew(usd_path)
assert False
UsdUtils.StageCache.Get().Erase(stage)
def test2():
UsdUtils.StageCache.Get().Clear() #this is what I would like to do
usd_path = "c:/temp/test123213.usd"
stage = Usd.Stage.CreateNew(usd_path)
assert True
The first test fails leaving a garbage stage in memory which affects the second one.
What I was after was a way to clear everything blindly before running a test not knowing what stages might have been created.
Again, this is a simplified example of a real case. According to the docs this should work: UsdUtils.StageCache.Get().Clear()
especially that: UsdUtils.StageCache.Get().Erase(stage)
does work. Maybe I’m not using it correctly? There must be a way to have a clean slate for each test to make sure they are properly isolated.
Reading through this thread, I wondering if there might be some confusion around stage caches and the layer registry.
Stage caches are optional, you must opt into using them. The layer registry is not. It’s a global singleton and all stages share layers in the registry. I’m wondering if the behavior your seeing isn’t related to stage caches, but rather the shared layer registry.
Reading through this thread, I wondering if there might be some confusion around stage caches and the layer registry.
It might very well be - but do you know then what API can be used to explicitly purge a particular layer from the shared layer registry?
Like a Sdf.Layer.Close(identifier) or similar?
Or maybe there are reasons that’s impossible and the intended approach to this is to actually reuse the layer if it exists in memory already? Along the lines of:
def create_stage(identifier):
layer = Sdf.Layer.Find(identifier)
if layer:
# open existing layer
# todo: potentially you may want to `clear` the layer here too?
return Usd.Stage.Open(layer)
else:
return Usd.Stage.CreateNew(identifier)
Then technically - what handle to that layer would still exist? Or what ways do I have to then explicitly flush out that layer’s identifier if I’m sure there is not existing reference to it? Is there any way force garbage collect it?
It currently sounds like even if there are no handles to it there’s no API to flush it?
The stickiness of the layer registry is something I’ve observed before and don’t have a great answer as to what’s going on.
I know that OpenUSD will both try to release the GIL and allows some objects to be destroyed asynchronously ( Universal Scene Description: pxr/base/work/utils.h File Reference (openusd.org)). It seems possible that either by design or by accident that one could reacquire the layer in the registry before it’s been fully cleaned up. I have not verified this is the cause. Just observing that there’s some complexity in deleting a stage.
works smashingly and without error for us in any python shell, and more generally, in cpython (which boost::python relies on, and therefore OpenUSD) should always gc the stage upon stage = None, and when the stage destructs, the layer will also be destroyed and removed from the layer registry (the file associated with it will remain on the filesystem, if using the ArDefaultResolver, but the next CreateNew() will happily overwrite it.
It seems clear, though, that folks are having issues, adn we’d like to help. Could someone provide us with a minimal, complete repro that demonstrates stage/layer lifetime problems?
I think my first oversimplified example is what created the most confusion. My apology. I also didn’t know there were two different caches, one for stages and one for layers. It seems that my problem is indeed related to the layer cache.
this is a simplified scenario that breaks for me in the second test:
from pxr import Usd, UsdUtils
class Test_usd():
def test_test1(self):
usd_path = "c:/temp/test123213.usd"
stage = Usd.Stage.CreateNew(usd_path)
assert False
def test_test2(self):
UsdUtils.StageCache.Get().Clear() # this is what I would like to do
usd_path = "c:/temp/test123213.usd"
stage = Usd.Stage.CreateNew(usd_path)
assert True
when the first test fails the stage is not released and makes the second test fail too. When the first test passes then the stage seems to be cleaned correctly and the second test executes fine. In my case running pytest in python 3.10 results with this unexpected error:
self = <scratch.Test_usd object at 0x000001FCBA299690>
def test_test2(self):
UsdUtils.StageCache.Get().Clear() # this is what I would like to do
usd_path = "c:/temp/test123213.usd"
> stage = Usd.Stage.CreateNew(usd_path)
E pxr.Tf.ErrorException:
E Error in 'pxrInternal_v0_22__pxrReserved__::SdfLayer::_CreateNew' at line 612 in file S:\jenkins\workspace\ECP\ecg-usd-build\ecg-usd-full-python3.10-windows\ecg-usd-build\usd\pxr\usd\sdf\layer.cpp : 'A layer already exists with identifier 'c:/temp/test123213.usd''
scratch.py:14: ErrorException
@BigRoyNL trick is neat but why do we need tricks like this in the first place?
We are wondering if pytest or another layer of your infrastructure might be hanging on to the exceptions raised by your tests? The backtrace associated with it would keep the raising frame alive, and with it all its locals, including the stage.
If that’s so, and you can’t change the policy, then unfortunately it would be on you to reset/del (should be equivalent) the stage before asserting.
You know what. I think I know. When testing this I quickluy tested this inside Maya with Maya USD. However, I believe that internally uses its own usd stage cache and it might be related to that?
However, that may make it extra weird that del stage did work?
