You get a file that renders fine in usdview and is almost 16GB (15,989,302,502 to be precise).
If you then try and usdzip that file, not even considering all the materials, just the animated geometry thusly:
usdzip stoat stoat.usd
You end up with a 3GB file (3,104,400,780 to be exact).
usdzip doesn’t report an error, and if I do the same steps from C++ checking for errors, I also don’t get any. usdview throws an error if I try and view it.
Is there something obvious I’m missing here (I do that a lot…)? If there’s not, I’m happy to try and dig into the code and see where the error is getting missed…
I seem to remember this coming up before (with usdcat?) but searching the obvious places didn’t turn anything up.
But essentially usdz is zip32 only. Our core specification will call this out explicitly.
However, the zipfile reader doesn’t read the table of contents when parsing a usdz, and instead reads all the individual file listings sequentially.
This has the confusing effect that as long as each individual file within a usdz is less than an unsigned 32bit int range , you can go above the zip32 total file size limit. But your file is now an invalid zip.
I consider this a flaw in the zipfile reader, and we should really fix the zipfile IO to be more valid. The core specification will specifically say the files must be zip32 compliant.
I suspect, though, that studios and others’ internal use-cases for usdz will eventually be unhappy with the 4Gb limitation… are we against broadening the spec to include zip64 when needed?
I think we’d all be fine with extending to Zip64 in a follow up spec version, unless Pixar feel that it’s something worth landing in 25.8 or the like? It’s a fairly trivial change imho but also not one I think folks are clamouring for immediately .
We’d agreed to put Zip32 in the first version of the spec since it’s what all USD apps can support today.
If we went over 4gb would there be repercussions? Part of the original motivation for the address space size was to accommodate memmap limitations that affected platforms of the time.
This would be very platform dependent. There’s platforms that would already not be fine with a 4GB file being memory mapped, but they’re also ones that wouldn’t be receiving such large files regularly.
I think it would be better to document that large files may not work well, but let people create it if necessary
Personally, I love the power of clear encapsulation that putting at least an asset in a USDZ gives you. I would totally want (as evinced by my question above) to be able to large assets with many variants in USDZ that was larger than 4GB.
Would be nice to revisit this, we are having issues regularly with the size of usdz and if it’s not resolved we will have to abandon USDZ and come up with something like USDBIGZ in our USD build.
Let’s concentrate any need/timing discussion on this GitHub Issue . I am curious about the severity and mitigations for this concern:
@nporcino , I don’t think we can leverage profiles to facilitate dynamic responses since you’d need to first open the file. But maybe it’s as simple as a file-size check before opening? Can we typically determine file-size prior to (e.g. in a web delivery context) downloading the file?
the response header from a HEAD or GET request usually includes (is expected to include) a Content-Length field. If the file is compressed, ie gzip(foo.usdz), the length will be the compressed length.
Basically we need an “atomic” format for USD that carries geo/textures etc. Most common use case is upload/download. Yes, we could user another package system and call it something else but it seems reasonable to just lift this limitation. For platforms not capable of dealing with these files sizes, a warning/exit seems a reasonable option that applied for situation where even 4GB is too big.