Spiff mentioned that we can accomplish custom serialization by developing ArAsset/ArWritableAsset/SdfFileFormat altogether. However, this approach is quite heavy, and we don’t actually need a new file format.
All we need to do is serialize USDC to a byte array and transfer it in memory or through web sockets. Since there are four years after I initially asked this question, we would like to know if there are any new and fancy interfaces that can help us achieve this purpose. Preferably, there is no need to develop SdfFileFormat.
I believe you should be able to accomplish this without needing a new FileFormat, now. You will still need a custom resolver (a URI resolver would be fine, and probably preferred) that whose OpenAssetForWrite() method returns an ArWritableAsset that puts data into the buffer you care about.
With that in hand, if you create your anonymous layer with a tag arg of “.usdc”, it should, I believe, create an in-memory crate-backed layer. Then, if you call Export() on the anonymous layer, with a “filename” that will resolve through your resolver, all the right stuff should happen. … this isn’t something we’ve actually tested, afaik.
A new resolver still sounds tricky since we have to specify “file formats” for those that are not “files.” And think about what happens in production. For fetching a byte array, we first have to instantiate a placeholder to mimic a “file” that is used for ArWritableAsset. Then, invoke the “Export” method, check if our placeholder has been filled, and extract the data from there.
We can treat it as a workaround, but I would like to know if there are possibilities to make what Matthew mentioned become the truth. When we first received this task, the very first thing that came to mind was also using ArAsset to read the buffer from the layer.
Sorry, @CalvinGu , I’m not understanding your reservations. I think what I proposed gives you exactly the workflow you described at the beginning of this post: you are starting with an anonymous layer in one process, and you are willing to Export it to a buffer that can be stably referenced or transmitted via socket. If you want the binary data to be usdc data, then you need to tell the anonymous layer that that’s the file-format encoding you want - don’t get hung up on terminology - we chose the name “FileFormat” twenty years ago when nothing other than files was in the picture, but it’s really just a “serialized data format” (SdfSdf?). The ArResolver you would right gives you the means to refer to such binary blobs in a standardized manner, and the ArWritableAsset lets any (Ar 2.0-upgraded) SdfFileFormat serialize/deserialize to it).
An anonymous layer itself, even in the default SdfData format, is not a binary blob, but rather a collection of C++ datastructures, and therefore not possible to send over a wire or treat like a buffer: that is precisely the serialization that Export() provides. So unfortunately, I don’t think what Matt suggested would give you what you need, even if anonymous layers could have resolvable asset-paths.
I agree there needs to be a placeholder to hold the serialized data, but I believe that can all be an internal detail of the ArResolver implementation, which could also have knowledge of the SdfLayerRegistry to aid in cache lifetime management of your blobs?
Thanks, Spiff. I understand that what you mentioned can fulfill our needs. And my last proposal was just exploring whether the workflow could be simplified.
Developing a resolver means mimicking the behavior of a file system. We have to make up an asset path and pretend to resolve it. We have to develop a corresponding “file system” on our side to receive the buffer from the resolver, which is kind of redundant.
Please allow me to illustrate this point using two cases:
In Houdini, we developed arHFs ArResolver to handle the “op:” file protocol. This helps USD import and export data from and to memory. However, “op:” is the Houdini built-in standard. Having a URL like “op:/folder/sub/name” makes sense for all Houdini developers. I could not agree more about using ArResolver in this case.
In Unreal, we are looking for an approach to serialize USD as a property of arbitrary objects for caching, communicating, and debugging purposes. In this circumstance, no file system or file protocol in Unreal can represent this requirement. We can make up a singleton URL like “ue:placeholder” to serialize USD to a fixed buffer in the memory. It just may not look like the most elegant solution for me.
Thanks again for your time, and I hope it can be helpful
… this isn’t something we’ve actually tested, afaik.