During large point cloud exports we have run into obscene amounts of runtime memory being consumed as each frame is written into the file. Each frame stair-steps the memory upwards until OOM.
However, we noticed that memory usage remains consistent, and low, if we call UsdStage::Save at the end of each frame, rather than just once at the very end of the process.
Is this the proper way to handle this situation?
Unfortunately we have encountered situations where calling Save multiple times on a single file is problematic (files are corrupt and do not contain all the data) for Google Drive (on Windows at least). Regular locations were ok, Microsoft OneDrive was ok, SMB shares were ok, but Google Drive was not. I suspect their filesystem filter driver is interacting very poorly with whatever Crate files are doing.
Hi Jesse,
Until you Save() any layer (usda-backed or crate-backed), the implementation is dutifully accumulating all the data you have authored into it, in memory, so we would expect the layer’s memory size to increase by almost exactly the size of the data you are authoring at each frame - is that what you are seeing?
Each time you call Save(), the crate layer deduplicates all the data it is holding, and flushes out “all the big stuff” out trhough the ArWritableAsset API (e.g. to the filesystem). So we do expect the memory footprint to drop significantly, and that is a practice we do in some of our “large data per frame” export scenarios when baking. There is a potential sacrifice in doing so: If you author some big data, call Save(), and then on a subsequent frame write the same data, we cannot deduplicate it against the earlier data, because it is no longer resident in memory to compare against. So, depending on the nature of your data, you might wind up with a larger file in the end, with more network traffic in some consumption scenarios. If your point clouds are the result of simulations, there might not be much duplication to take advantage of, anyways.
Please do not attempt to repeatedly Save() crate files directly to gdrive, though. It’s nothing to do with crate, per se, but gdrive is 100% not designed to handle machine-speed “append some more data to a file” operations (which is all crate is doing), and is, by google’s own documentation, a recipe for corruption. It’s fantastic for human-speed continuous updating with live-updating for multiple clients, which is what it’s designed to do in support of gsuite.
Yeah, the size increase is proportional, if not exactly, the amount of data that’s being authored. I guess I had naively assumed that de-duplication would only need to keep at most 1 frame (the prior set of values) in memory and not everything up to that point; though I didn’t really think to much about it before.
So it looks like we’ll have to engineer in a way to allow users to invoke save multiple times with relevant explanation of the pros/cons of that approach.
And just so it’s clear, the user that hit this issue with gdrive wasn’t expecting anything close to realtime replication to other clients - at least I don’t think they were. It was most likely half used as a general purpose backup and half as an easy way to work across machines for their own use between office and home. I guess we were all very surprised that gdrive was not eventually consistent over any time scale and was actually just foo-bar’d entirely.
Understood. My takeaway (learned in response to this thread) which may not be completely accurate, but as maybe a rule of thumb for uses with USD, at least, is to think of gdrive exactly as a backup system, or like a drop-box, and not as an alternative to NFS or local disk.