Binary SDF file size doe not decrease on content removal

Community USD
1

Hello!

I have noticed that when removing content from a binary SDF layer and saving that layer, the file size does not shrink on disk. I assume this is by design to make serialization quicker on the assumption that you would rarely do such a thing? Is this right?

I have a use case where I do need to remove content from layers (this is how i noticed this) - a workaround I have found is to export the layer to another binary layer and then swap the files (exporting ‘‘in place’’ doesn’t work, i think it sees it’s the same layer, sees its not dirty, and doesn’t do anything).

Wanted to confirm this is by design, and was wondering if there is a better way to handle this that I maybe missed? Did I miss a util or api to force it to fully reserialize?

Thanks in advance!

2

USDC files are designed to allow appending new information to the end of the file without having to re-write the entire file (which might be many gigabytes in size). So you can “delete” data from the SdfLayer (and/or add a few new attributes or prims), save the modified file, and only a few kilobytes of new data get appended to the end of the file. The many GB of preceding data is not re-written. Huge performance win. At the cost of some “dead” disk space.

The save-to-another-file-then-move is the right way to do this when you have decided that the time cost of writing the complete file is worth it for the amount of disk space you’ll be saving. I’m not sure if there is a built-in SdfLayer API for this, but I think it would be nice if there was. Seems like it would be a common thing for one to want to do.

3

Yes that makes sense.

Indeed it would be useful, maybe another optional flag to the save (it already has ‘‘force’’ to bypass the dirtiness check)

Thanks for your answer!

4

A trim flag would be nice, I agree.

5

A “trim” flag would be a little overly-format-specific to add to SdfLayer::Save() (and UsdStage version). @amohr reminded me that we’d had earlier discussions about someday, e.g. if crate supported any kind of lossy compression, that the way we’d provide controls for that would be to have an RAII context object you could have on the stack, which would then affect all saved/exported crate files. Seems like a trim flag would fit in there as an option, also, and keep it out of everyone else’s hair.

1 Like
6

That makes sense. Or perhaps arguments in a dictionary to the save/export call.