You get a file that renders fine in usdview and is almost 16GB (15,989,302,502 to be precise).
If you then try and usdzip that file, not even considering all the materials, just the animated geometry thusly:
usdzip stoat stoat.usd
You end up with a 3GB file (3,104,400,780 to be exact).
usdzip doesn’t report an error, and if I do the same steps from C++ checking for errors, I also don’t get any. usdview throws an error if I try and view it.
Is there something obvious I’m missing here (I do that a lot…)? If there’s not, I’m happy to try and dig into the code and see where the error is getting missed…
I seem to remember this coming up before (with usdcat?) but searching the obvious places didn’t turn anything up.
But essentially usdz is zip32 only. Our core specification will call this out explicitly.
However, the zipfile reader doesn’t read the table of contents when parsing a usdz, and instead reads all the individual file listings sequentially.
This has the confusing effect that as long as each individual file within a usdz is less than an unsigned 32bit int range , you can go above the zip32 total file size limit. But your file is now an invalid zip.
I consider this a flaw in the zipfile reader, and we should really fix the zipfile IO to be more valid. The core specification will specifically say the files must be zip32 compliant.
I suspect, though, that studios and others’ internal use-cases for usdz will eventually be unhappy with the 4Gb limitation… are we against broadening the spec to include zip64 when needed?
I think we’d all be fine with extending to Zip64 in a follow up spec version, unless Pixar feel that it’s something worth landing in 25.8 or the like? It’s a fairly trivial change imho but also not one I think folks are clamouring for immediately .
We’d agreed to put Zip32 in the first version of the spec since it’s what all USD apps can support today.
If we went over 4gb would there be repercussions? Part of the original motivation for the address space size was to accommodate memmap limitations that affected platforms of the time.
This would be very platform dependent. There’s platforms that would already not be fine with a 4GB file being memory mapped, but they’re also ones that wouldn’t be receiving such large files regularly.
I think it would be better to document that large files may not work well, but let people create it if necessary
Personally, I love the power of clear encapsulation that putting at least an asset in a USDZ gives you. I would totally want (as evinced by my question above) to be able to large assets with many variants in USDZ that was larger than 4GB.