Like the UsdStage::ExportToString() that creates an in-memory usda, it would be great if there was an ExportToUsd() that would populate a buffer with binary usd data. It would make transforming usd data on the fly much easier. Right now, we are serializing to disk and then reading that file into memory to work around this.
Hi @mobright , you’re thinking about exporting, so the naming is not obvious and discoverable, but what you are asking for is exactly what UsdStage::Flatten() does. Un;ess I’m misunderstanding what “reading that file into memory” means?
Hrm. I must be asking this wrong. We want to export a stage with just the “proxy” prims after loading all payloads. We currently open the stage with “load all” enabled. Delete all the leaf non proxy prims. Flatten. Now we need to get that into a buffer to send to over a websocket.
If we called ExportAsString, the resulting buffer is almost a gig. If we write that to a file with export, that buffer is 170M.
Is there a way to get the usdc version of a stage into an in memory buffer?
MO
Ah, I see. This has been coming up several times now. @CalvinGu has been looking for an easier way for awhile now, and @mwestphal has also been trying to solve the more general version (I think) of what you’re trying to do (more general because if you are flattening the stage before serializing it, the “references to external layers” problem doesn’t arise).
You can get the latest guidance we have in that latter forum thread. You are asking only about getting the usdc encoding of a layer into a buffer without passing through any filesystem. But can I ask what your expectation is on the other side of your network transmission? Is it the same as @mwestphal in that you then want to reconstitute a layer and put it on a stage? If so, what would your expectation be if that layer has references to other layers that may not exist on the other side, or textures, etc? I ask because the “export to buffer” seems easier and less fraught for us to support in a core way, but like I described in the other thread, we think there’s potential for breakage and confusion if we facilitate just creating an SdfLayer (even anonymous) from just a buffer.
1 Like
We want to display a somewhat large usd stage on a user’s machine in the web browser (using the enscripten build of usd). Our asset structure has a proxy, render, and solid brep representation in it and needs a custom arresolver to piece it together. On the user’s machine we only need the proxy version for display. We looked into streaming all of the files over to the user’s browser, but the number of files is massive and the space includes the representations we don’t need, which seemed silly.
Hence the delete and flatten process which gets us to a singular file, doesn’t require a custom resolver, and is the minimal amount of data we can get to.
I would be fine with a call that takes the layer and creates a usdc buffer and keeps the references as references. It would be something we would look into and potentially use. The ExportToString is the type of signature we want, but the data size difference between ascii and binary of mesh data is a killer.
I could see a world where replacing our arresolve uri’s with relative paths and streaming the payload files so they are shared might be another option, but it would currently require rewriting each asset to disk and our stage composition layers as well. All of which would benefit from an in-memory version instead of requiring serialization.
Thanks, Michael, that makes sense. Just to zone in one the core request/requirements, here:
- So the server process that is loading and flattening a UsdStage is also a web-server, serving the serialized data to the client?
- How is that stream handled on the web client end without a custom resolver to transform it back into a SdfLayer? (E.g. given the problems @mwestphal noted in trying to do just that)
Thanks!
–spiff
There is a web-server that does have our custom usd, custom arresolver, and other usd plugins. It’s job to is to create a flattened usd that can run in a simplified web version of usd, on the client, without our custom plugins. After creating that stream, it sends it to the client via a websocket.
The client pulls the flattened file, saves it to an emscripten virtual file system on the client, and uses hyrda → webgl to display that usd hiearchy on the client. The client has an emscripten compiled version of usd including the default arresolver, so it could potentially reference things using the default arresolver.
The issue we are trying to figure out is that our asset structure contains versions of the geo we don’t need on the client. We want the most efficient way to strip those out. Right now we do that by loading the stage and all models, deleting the prims we don’t need, flatten, serialize (to get usdc), free the stage, read the file, send the file. I was hoping to do those last steps in memory without the serialize step.
Thanks, Michael - that helps alot, and what you’re doing is something I’d expect USD servers to want to do, and worthwhile for the core API’s to facilitate, on the server-side; and as I mentioned before, in a similar space to something we’ve heard from multiple parties now. Please stay tuned.
Upon a first, brief discussion, there was questioning of what the technical problems with writing out to the filesystem impart to your server, given that:
- Adding new
ExportAPI to SdfFileFormat and SdfLayer comes with some challenges, especially given the number of FileFormat plugins that do not support the ArWritableAset abstraction, which would be needed, here. - A belief that, performance-wise, the serialization to disk should not impart a significatn cost, given that all the pages will be hot if you’re going to just turn around and slurp the bytes back up, and it may give you lower overall memory pressure.
We haven’t actually verified that second claim, but we’re curious if there are other sandboxing, access, permissions, maintenance/coordination, etc. issues that make filesystem writing impractical.
Thanks!
It’s just a point of failure and additional steps to something that doesn’t really need extra steps.
The server might run out of disk space, the server might have perm problems, the server needs to be deployed with enough space to perform just this operation. You need to make the server is writing to a unique file if you are threading/forking this.
So, it’s not holding me up in that I have a functional workaround. It more just complicates things. I was hoping since the ExportAsString existed a similar call to ExportAs binary would be easy.
MO
Hey Michael, following up… we spoke to some other clients with similar needs/concerns, and they thought a utility API that took a stage (or layer?) and returned you a buffer would work for them, which behind the scenes would:
- Using Arch API’s, created a temp file to export to…
- exporting the stage/layer
- mmap the asset it back in
- return a pointer to that mapped memory, after
- unlinking the file
That doesn’t address all your concerns, as it could be possible that the server can’t even create tmp files, but it prevents tmp files from accumulating and clogging your server.
Would that be usable/useful in your situation?
Yeppers. Sounds good.