How can we ensure consistent AssetBundle builds across multiple machines, and avoid large re-downloads in case of machine or disk failure?
Currently, hot update builds for different platforms are distributed across multiple machines. Considering disaster recovery scenarios, if a build machine or its disk fails, rebuilding may force players to re download several GB of hot update content. Beyond periodically backing up the entire project or the Library folder, are there better solutions? Additionally, how can we ensure that different machines and environments produce identical AssetBundle outputs, assuming Meta files are already under version control?
To ensure consistent AssetBundle AB builds across multiple machines in different environments or regions, and to mitigate risks caused by machine or disk failures, a systematic design is required across three key dimensions: build consistency, cache management, and disaster recovery . 1. Ensuring Build Consistency Unified Build Environment All build machines must use exactly the same environment , including: Unity version Operating system patches JDK / Python / SDK dependencies To eliminate environment drift, it is recommended to standardize environments using: Docker images, or Virtual machine templates This ensures that differences in GUID generation or serialization behavior do not occur due to environment inconsistencies. Shared and Locked Library Folder Cache data inside the Library folder such as , , etc. has a direct impact on AssetBundle generation results. Recommended approach: Treat critical cache directories as read only Distribute them through: Network storage e.g., NFS , or CI/CD pipeline artifact distribution This ensures all build machines rely on identical intermediate build outputs. Disable Non Deterministic Factors Ensure deterministic builds by disabling or controlling sources of randomness, such as: Random seed variation Timestamp embedding in build outputs If using Unity Addressables, ensure: Build scripts are identical across all machines Content providers and configuration files are strictly version aligned 2. Build Caching and Incremental Reuse Centralized Build Cache Service Use one of the following: Unity Cloud Build Cache Self hosted artifact systems e.g., Artifactory / Nexus Store: Successfully built AssetBundles Associated metadata hashes, manifests When a build machine fails, historical builds can be directly restored from cache without rebuilding from source. Incremental Build Strategy Use AssetBundleManifest to compare differences between versions and only rebuild modified assets. Combined with differential patching via CDN mechanisms e.g., BSDiff style updates , this approach: Minimizes hot update package size Enables fast recovery even after full rebuilds 3. Disaster Recovery and High Availability Distributed Backup System In addition to regular project backups, every successful build should be synchronized in real time to at least two geographically separate storage systems, such as: Cloud object storage S3 / OSS Also enforce: Version retention policies This ensures historical builds are always recoverable. Automated Failover Pipeline Implement a build cluster health monitoring system: If a primary build machine fails: Automatically trigger a backup machine Rebuild from cache or source Validate output consistency via hash comparison You can integrate validation tools such as Unity based AssetBundle verification utilities e.g., UWA tools to ensure resource integrity. Content Addressable Storage CAS Adopt a content addressable storage system , where each AssetBundle is stored and identified by its content hash. Benefits: Same input always produces interchangeable outputs Eliminates dependency on machine paths or timestamps Ensures build reproducibility across all environments Summary By combining: Standardized build environments Shared intermediate build artifacts Centralized caching systems Multi region backup and failover mechanisms It is possible to: Guarantee consistent AssetBundle outputs across machines Rapidly recover from single node failures Minimize redundant player downloads and hot update risks This architecture ensures both build determinism and production level resilience for large scale Unity hot update pipelines.