Discussion: Deprecation of HDF5 support in UsdAlembic plugin

Community USD
1

Dhruv has mentioned in the ASWF vfx reference platform slack channel, that the Alembic HDF5 backend has been dropped from macOS distribution for some years, and that maintenance, security, and performance concerns, prompt a desire to eliminate support for it entirely in favor of supporting only the Alembic Ogawa backend.

Further to his points, I have elaborated some background here with a proposal for your consideration.

Alembic has marked HDF5 as optional not required, ever since the introduction of Ogawa.Hereā€™s the original notice from Steve Lavietes https://groups.google.com/g/alembic-discussion/c/FTG1HuuO_qA The reasons given then for moving beyond HDF5 were

With a few years of experience with Alembic now, itā€™s clear we need a back-end data format that meets the following requirements:1) A simple API to easily and efficiently express the higher-level Alembic APIs without change to client code.
2) Minimal library dependencies
3) Data sharing for Alembicā€™s de-duplication features
4) Input/output abstraction (for potentially reading from non-file sources)
5) Thread-efficient reading
6) Low overhead for small data cases

The immediate benefits were measured as

  1. File sizes are on average 5-15% smaller. Scenes with many small objects should see even greater reductions.
  2. Single-threaded reads average around 4x faster
  3. Multi-threaded reads can improve by 25x (relative to the same operations in the existing HDF5 back-end) on 8 core systems.

HDF5 support is further identified as implementing backwards compatibility.

Of course, weā€™ll maintain backwards compatibility. Backwards compatibility remains another key goal of the Alembic project and as such, the ability to read and write Alembic data with the existing HDF5 back-end will continue to be supported for the foreseeable future.

He goes on to say

Itā€™s possible to build without HDF5 but you sacrifice the ability to read older archives.

Users then go on to report results

Using HDF5 we were not able to get more than 1 fps (around 0.8), now we can play it at more than 20 (23.8 average) !

I cannot find evidence that the Alembic team has publicly discussed deprecating or removing HDF5, so the above is not meant to imply that Steve or anyone else thinks or thought HDF5 as an alembic backend should be discarded. I nonetheless present it as evidence that HDF5 was discovered to be inadequate to function as an Alembic backend from an early date, and thus Ogawa became a very successful successor.

I present the above as support for the idea that HDF5 for Alembic be removed from the reference platform moving forward. Additionally, I take a strong position that HDF5 not be a supported back end for Alembic in USD, unless a user builds the Alembic plugin themselves. A user sophisticated enough to need Alembic with HDF5 also has the sophistication to build the Alembic plugin and HDF5 itself.

Further, my strong corollary position is that Alembic with HDF5 not be a ā€œuniversal archival targetā€ for use as a referenced layer in USD.

I invite discussion on this topic.

1 Like
2

In further support of removing HDF5 support from USD - the NVIDIA built USD binaries - both the standalone binary, and the one included with Omniverse - dropped HDF5 support a bit back as well, for security reasons. (Rough guess is it happened > 1yr ago.)

If there was any complaint or backlash over this, it never reached my earsā€¦

3

Similarly, we dropped HDF5 as well on macOS as well, and didnā€™t hear any complaints.

For reference, I put up this PR that has two commits. One that removes HDF5 from the build_usd.py, and the other that removes it completely from the codebase.

I figure that way we can easily discuss what form deprecation or removal looks like, and the work is already done regardless of the route folks decide.

My one fear is that if we only remove it from the build usd.py, we have a high risk of bit rot since very few people would use it at all (assuming they are now).

I think at that point, if the support is left in the codebase, it should be made clear that itā€™s deprecated and community maintained? I would suggest that whatever company or individuals advocate for keeping it would then be point persons for testing and maintaining any regressions.

This brings me to the point that I think USD should perhaps have documented support tiers for features and platforms.

4

I added thoughts from SideFX to the PR on github.

TL;DR is that weā€™d appreciate it if the C++ code remained in place to minimize differences in our USD fork, but one way or another SideFX will continue to support reading HDF5 Alembics (in Houdini in general, and therefore in our USD library as well) for at least a little longer.

1 Like