How to Access JuiceFS via S3 and WebDAV Protocols
Different applications access data in different ways: some rely on POSIX file interfaces, some are built on the S3 API, and some users want to browse and edit remote files through a file manager. JuiceFS' multi-protocol access capability is designed precisely to let these tools access the same file system.
In this article, we'll explain the shared initialization path of S3 Gateway and WebDAV, describe their applicable scenarios and common mappings, and dive into their implementation principles and boundaries.
A shared initialization path
JuiceFS consists of a metadata engine and object storage:
- The metadata engine manages directories, attributes, locks, and sessions.
- Object storage stores the actual file data blocks.
!Image description
From an implementation perspective, S3 Gateway and WebDAV first go through the same service initialization path. The service process creates a JuiceFS client based on the metadata address, listen address, and common parameters such as subdir, read-only, cache, limits, access-log, and metrics. Then, it initializes the metadata client, object storage, data block cache, and VFS configuration in sequence, ultimately obtaining an fs.FileSystem.
!Image description
In other words, the common foundation of the two is not a FUSE mount point, but a native JuiceFS client with sessions, caching, metrics, and background tasks. Before S3 requests and WebDAV requests enter their respective protocol adaptation layers, they already share the same JuiceFS file system and the same file system namespace.
Applicable scenarios and common mappings
The shared initialization path explains how the two connect to the same file system. In actual use, what matters more is choosing the access method: which scenarios are suitable for S3 Gateway, which are suitable for WebDAV, and how common requests map to JuiceFS operations.
- Linux applications, training jobs, or workloads requiring full file system semantics should prefer FUSE/CSI.
- Applications already built on the S3 SDK, AWS CLI, MinIO Client, or object storage toolchains should prefer S3 Gateway.
- Desktop file manager access, DAV/rclone synchronization, Office/apps collaboration, and simple sharing should prefer WebDAV.
Mapping of common S3 Gateway APIs to JuiceFS file system operations:
| S3 API | Purpose | Equivalent file system operation |
| :---- | :---- | :---- |
| GetObject | Download an object | open/read |
| PutObject | Upload or overwrite an object | create/write/rename |
| HeadObject | Query object metadata | stat/getxattr |
| DeleteObject(s) | Delete one or more objects | unlink/remove |
| ListObjects V1/V2 | List objects by prefix | readdir+filter |
| CopyObject | Copy an object server-side | copy/read+write |
| MakeBucket | Create a bucket | mkdir |
| DeleteBucket | Delete an empty bucket | rmdir |
| NewMultipartUpload | Create a multipart upload session | create |
| PutObjectPart | Write a part | write |
| CompleteMultipart | Merge and commit the object | copy + atomic rename |
| Tags/Metadata | Read/write tags and metadata | get/setxattr |
Mapping of common WebDAV methods to JuiceFS file system operations:
| HTTP method | Purpose | Equivalent file system operation |
| :---- | :---- | :---- |
| GET | Download a file | read |
| PUT | Upload or overwrite a file | write/create |
| DELETE | Delete a file or directory | remove |
| MKCOL | Create a directory | mkdir |
| PROPFIND | Query properties or list a directory | stat/readdir |
| PROPPATCH | Modify custom properties | setxattr |
| COPY | Copy a file or directory | cp |
| MOVE | Move or rename | mv/rename |
| LOCK | Lock resources | File lock |
| UNLOCK | Unlock resources | Release a file lock |
| OPTIONS | Query server capabilities | Capability negotiation |
| HEAD | Query HTTP metadata only | Lightweight stat |
S3 Gateway principles: from object semantics to file system operations
S3 Gateway converts S3 protocol requests into JuiceFS file system operations. This section covers four aspects:
- The request path
- Internal state
- Key semantic conversions
- Permission coordination
Request path: from the MinIO S3 server to jfsObjects
The MinIO S3 server handles the S3 protocol layer, including request routing, signature verification, authentication and authorization, and multipart upload orchestration. After requests pass through the protocol layer, they enter the ObjectLayer implemented by JuiceFS, namely jfsObjects. jfsObjects converts object-level reads, writes, listings, deletions, and attribute access into JuiceFS file system calls.
!Image description
State storage: how the internal namespace carries the Gateway state
During request conversion, the Gateway does not only access user-visible object paths; it also uses an internal namespace to store state needed by the service itself. For example, .sys/tmp is used for temporary writes, .sys/uploads is used for multipart upload state, and .minio.sys is used to store MinIO/IAM-related metadata.
These internal paths are not application data in the ordinary user's understanding, but they are the foundation the Gateway needs to maintain upload processes, part states, and permission configurations.
Semantic conversion: PutObject, Multipart, Bucket, and Metadata
The S3 object model and the file system model are not completely the same, so the Gateway needs to perform semantic conversions at several key points.
PutObject must ensure that an object is visible only after the upload is complete. To prevent clients from reading incomplete objects, JuiceFS S3 Gateway writes to a temporary file and saves object attributes such as metadata and tag, commits to the target path via rename after completion. Leveraging the atomicity of file system rename, the Gateway can switch an object write from “uncommitted” to “committed.”
!Image description
Multipart upload needs to merge multiple parts into the final object. JuiceFS S3 Gateway can use file system capabilities such as CopyFileRange to combine already written parts and commit the final file via rename. This mainly adjusts file system metadata relationships rather than moving all data again.
!Image description
Bucket mapping handles the relationship between the object namespace and file paths. In single-bucket mode, the file system name corresponds to the bucket; in multi-bucket mode, top-level directories map to buckets. Object attributes such as metadata, tag, and ETag are stored through file extended attributes (xattr).
Permission coordination: IAM and multiple instances
S3 Gateway also needs to handle identity and permission information, including root users, regular users, policies, service accounts, and more. These configurations are stored in the JuiceFS file system through internal paths, allowing them to be reused across multiple Gateway instances.
Therefore, multiple Gateway instances can share the same set of IAM configurations; however, each instance caches IAM information, so permission changes are not synchronized instantaneously. In actual deployments, we need to consider the refresh cycle.
WebDAV principles: from file protocol to JuiceFS operations
WebDAV starts from a different point than S3 Gateway. It targets file access scenarios, with common clients including macOS Finder, DAV/rclone, and some Office/apps clients. Since the WebDAV protocol includes concepts such as files, directories, and attributes, it does not need to first handle the model conversion between buckets, objects, and keys on the one hand and paths and files on the other, unlike S3 Gateway.
Request path: from HTTP entry to webdavFS
After a WebDAV request enters JuiceFS:
1.It passes through net/http and indexHandler. This layer mainly handles HTTP entry-related capabilities, such as authentication, transport, and conversion of directory access requests.
2.The request enters x/net/webdav.Handler, which handles the WebDAV protocol layer and converts protocol methods into file access requests that webdavFS can process.
!Image description
3.After entering webdavFS, the file access requests passed in by the Handler are converted into JuiceFS file system calls such as open, create, read/write, rename, delete, and attribute setting. Since the WebDAV protocol is close to file system semantics, the focus here is not to rebuild an object model, but to handle how request methods, file open flags, and attribute states map onto JuiceFS operations.
State boundaries: dead properties and LOCK
WebDAV adaptation involves not only data reads and writes but also the protocol's own state. Dead properties can be saved as JSON in xattr, thus landing in JuiceFS' file attribute system; however, LOCK state is different—it's currently kept in the service process memory. Single-instance deployments usually have no problem; in multi-instance deployments, different instances cannot automatically synchronize lock state.
These limitations usually do not affect Finder browsing, simple sharing, or DAV/rclone synchronization. However, if an application strongly relies on WebDAV file lock semantics, deployment mode and client behavior need to be verified separately.
S3 Gateway and WebDAV share the same JuiceFS client initialization path and access the same file system namespace. The difference is that S3 Gateway needs to convert object semantics into file system operations, while WebDAV is closer to file access semantics.
Therefore, the choice of access method should ultimately depend on the application itself: whether it organizes data as objects or operates on data as files and directories.
If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.
1 reaction
Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s
In large-scale AI inference services, when a cluster scales out, newly added inference instances must go through a series of cold-start steps before they can serve requests: container image preparation, file system mounting, runtime and inference framework initialization, and model weight loading. As image sizes and model weights continue to grow, the time spent on data preparation becomes increasingly prominent. During large-scale scaling events, many instances start simultaneously, further increasing the concurrent pressure on the registry, network, and backend storage.
To address these issues, the industry has developed several optimization strategies, including reducing image size, using P2P distribution to alleviate concurrent pull pressure, and employing on-demand loading to reduce data transfer on the critical startup path.
In this article, we’ll explore a shared‑storage‑based approach: combining Nydus’ on‑demand image loading capabilities with JuiceFS’ distributed storage and caching mechanisms. By sharing storage and reusing cached data, this approach reduces redundant data reads during large-scale scale-outs and shortens container startup time. For AI inference scenarios, container images and model weights can be managed together under a unified storage, caching, and prefetching system, accelerating both types of data while reducing the complexity of building and maintaining two separate data pipelines.
Why OCI image distribution is slow
Traditional Open Container Initiative (OCI) images are typically organized as compressed tar layers. In the image‑pull mode, a node must fully download and extract the required layers before the container can start. In practice, however, only about 6%–20% of files in an image are actually accessed during container startup; the rest of the download and extraction overhead is wasted on non‑critical‑path data.
In large clusters, this “prepare everything first, then start” approach causes three problems:
- Long cold‑start times: For GB‑sized images (like AI inference images), full download and extraction significantly delay new instance startup, often taking minutes.
- Registry bottleneck: During HPA‑triggered bursts or large‑scale rollouts, hundreds of nodes simultaneously request GB‑sized layers from the registry, easily overwhelming the registry and congesting the network.
- Redundant network and storage usage: Without an efficient P2P or distributed sharing mechanism between nodes, the same layers are repeatedly transferred across the cluster, resulting in significant bandwidth consumption and duplicated storage.
Dragonfly+Nydus: what it solves and what it leaves behind
To mitigate registry bandwidth pressure during concurrent image pulls in large clusters, a common solution is to introduce P2P distribution systems like Dragonfly. The core idea is to build a P2P distribution network via a scheduler and node‑side agents, allowing nodes to share image data and reduce egress traffic from the registry.
However, P2P distribution primarily optimizes the data transfer path; it does not change the fundamental OCI mechanism of downloading and extracting layers before starting the container. Nydus, on the other hand, changes the loading approach: it transforms “prepare everything then start” into “fetch lightweight metadata first, and then read actual data on demand,” reducing data transfer on the critical startup path.
Combined, Nydus handles on‑demand loading, while Dragonfly distributes the on‑demand data, reducing both the actual image read volume and the concurrent access pressure on the registry.
However, this combination introduces additional infrastructure and operational complexity. Besides Nydus snapshotter, one must deploy and maintain Dragonfly’s scheduler, distributor, and node‑side components, and handle P2P network connectivity, cache space management, node anomalies, and cache eviction. In large‑scale GPU inference clusters, this means maintaining a separate data distribution and caching system independent of the storage layer.
This leads to an alternative thought: if the cluster already has a mature distributed file system like JuiceFS, could image data distribution and caching be offloaded to the shared storage system, reusing existing storage infrastructure while reducing the cost of building and maintaining extra P2P components?
Architecture redesign: Nydus blobs hosted by JuiceFS
The core idea of the Nydus+JuiceFS solution is to transform image data distribution—originally relying on P2P network scheduling—into shared storage and cache reuse based on a distributed file system. In this architecture, JuiceFS serves as the underlying unified data engine hosting Nydus image blob data, and worker nodes read data through the standard POSIX interface.
!Image description
After an image is built and converted to Nydus format during the CI/CD phase, it’s split into two parts:
- Bootstrap (metadata): Very small (typically a few hundred KB, for example, ~450KB), stored in the Image Registry, used for fast indexing.
- Blobs (actual data blocks): Contains all file content (typically several GB, for example, ~1GB), persisted directly to the JuiceFS shared file system after conversion (for example, /var/lib/nydus/blobs).
When a worker node starts a container, containerd triggers nydus-snapshotter, and nydusd mounts the container’s read‑only layer via FUSE. Then, when the application issues actual read requests, nydusd looks up the index in the Bootstrap and reads the corresponding blob data on‑demand through the JuiceFS mount point.
The key change in this path is: the registry only distributes the manifest and the extremely lightweight bootstrap; the massive blob data transfer is entirely handled by JuiceFS’ distributed caching system. This fundamentally eliminates the registry network bottleneck during high‑concurrency pulls.
For AI inference scenarios, this architecture can further unify the storage paths for images and model weights. In addition to Nydus image blobs, model weights can also be stored in JuiceFS, allowing both types of data to reuse the same storage, caching, and prefetching mechanisms. This accelerates data loading while reducing the complexity of building and maintaining two data pipelines.
| Comparison basis | OCI full image pull | Dragonfly+Nydus | JuiceFS+Nydus |
| :---- | :---- | :---- | :---- |
| Core concept | Full layer pull + layer extraction | P2P network offloading + on‑demand image loading | Storage‑layer decoupling + distributed caching + on‑demand loading |
| Cold start method & time | Full download and extraction required; typically 30s to 10min+ | Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds | Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds |
| Image data transfer path | Registry → worker (full local download & extraction) | Registry / source → Dragonfly P2P network → worker on‑demand read | JuiceFS → worker on‑demand read via POSIX, accelerated by local & distributed caches |
| Core components | Registry + containerd | Nydus, Manager, Scheduler, Seed Peer, and dfdaemon | JuiceFS + nydus‑snapshotter |
| Image & model weight synergy | Image pulled from the registry to local node; model weights loaded via a separate path (NAS or object storage); two systems managed independently | Images loaded on‑demand via Nydus, distributed by Dragonfly; model weights still rely on independent external storage | Image blobs and model weights stored together in JuiceFS, sharing the same cache and warm-up mechanisms |
| Operational complexity | Easy to overwhelm registry bandwidth under high concurrency | Requires extra maintenance of P2P scheduler, node network, and caching system | Can reuse existing JuiceFS storage & cache infrastructure, reducing independent image distribution components |
| Local disk pressure | High: Stores full image and extracted data | High and unpredictable: Requires cache space for P2P; rapid scaling may cause disk full | Controllable: Unified shared storage pool, uses local NVMe for P2P distributed cache |
| Network requirements | Standard TCP | Requires widely open P2P ports across nodes; may be invalid in cross‑VPC/NAT environments | Standard LAN – only requires normal high‑bandwidth inter‑node LAN; no intrusive network requirements |
| Suitable scenarios | General workloads with no cold‑start requirements and small clusters (<20 nodes) | Ultra‑large clusters across data centers / clouds / edge computing environments without unified high‑bandwidth storage; strong ops capabilities of the team | Ultra-large clusters across data centers / clouds / edge computing environments without unified high-bandwidth storage, where JuiceFS is already deployed or can be deployed, aiming for unified image+model management and simplified ops |
Runtime mechanism: how Nydus and JuiceFS coordinate
Startup phase: fetching bootstrap (metadata)
When the application starts, nydusd only needs to fetch the lightweight bootstrap file tree from the registry. The bootstrap describes the image file tree and block indexes, but contains no actual file content.
After the container root filesystem (rootfs) is mounted, the subsequent startup process proceeds. In our test scenarios, containers can be ready and pass health checks within hundreds of milliseconds, without transferring the full large image data before startup.
Read phase: on‑demand blob access
When the application process actually issues read operations (such as open() or read()), nydusd captures the I/O requests via the Linux FUSE mechanism and locates the corresponding blob data block and offset using the bootstrap index.
This process is completely transparent to upper‑layer applications. For the underlying data path, the read granularity changes from full image layers to the data blocks corresponding to specific files and offsets. nydusd directly accesses the JuiceFS mount point (for example, /var/lib/nydus/blobs) to read the target block, completely eliminating the runtime fetch‑from‑registry path.
Cache phase: data reuse via JuiceFS cache groups
JuiceFS’ distributed cache groups can reuse already‑cached blob data across the cluster. On a cache hit, data is returned to the kernel at near‑local‑NVMe speed. On a cache miss, only the specified offset block is pulled on‑demand and automatically filled into the cache. In addition to the automatic fill‑on‑miss, blobs can be pre-loaded into the cache to ensure full cache hits during on‑demand reads.
For AI inference scenarios, the same mechanism can also be applied to loading model weight files.
Production test data
In production, we compared the solutions above against a large AI inference image.
Container startup time comparison
The table below shows the test results for container startup time:
| Scenario | OCI overlayfs | Nydus+JuiceFS | Comment |
| :---- | :---- | :---- | :---- |
| Cold pull + run | 116 seconds | 16 seconds | OCI uses public cloud registry; Nydus+JuiceFS uses JuiceFS + public cloud object storage |
| Cold pull + run (with distributed cache warm-up) | N/A | ~1.4 seconds | Required blobs pre‑loaded in JuiceFS distributed cache |
| Warm pull + run | ~0 seconds | 0.22 seconds | OCI: same tag already on node; Nydus+JuiceFS: blobcache local cache |
From the test results, we can see:
- 7x speedup without cache warm-up: When neither local nor distributed cache is warmed up, Nydus+JuiceFS reduces container startup time from 116s to 16s.
- ~99.9% reduction in data read volume during startup: In the cold pull + run scenario, read volume drops from 11 GB to 11 MB. Traditional OCI requires full download and extraction before startup, while Nydus+JuiceFS only fetches bootstrap metadata and reads blobs actually accessed during startup.
- 1.4s startup with distributed cache warm-up: When blobs are cached in JuiceFS distributed cache, startup takes 1.4s; with local blobcache warmed up (warm pull + run), startup takes only 0.22s.
On‑demand file read performance
The table below shows test results for: on-demand file read performance:
| Scenario | Traditional OCI (local disk direct read) | Nydus+JuiceFS (distributed cache hit) |
| :---- | :---- | :---- |
| First‑block read latency (TTFB) | ~0.9 milliseconds | ~3.2 milliseconds |
| Small file / scattered configuration loading (IOPS) | Baseline | Close to baseline |
| Large file sequential read throughput | Local disk limit | GB/s level (JuiceFS NVMe distributed cache) |
From the test results, we can see:
- Time to first byte (TTFB): On distributed cache hit, first‑block read latency is ~3.2 milliseconds, about 2.3 milliseconds higher than local disk direct read (0.9 milliseconds). The extra latency comes from the FUSE data path and cache access, but remains at millisecond level, generally imperceptible to the application side.
- Small file IOPS: Performance is very close to local disk baseline. In scenarios with dense Python dependency and configuration file reads, JuiceFS caching introduces no noticeable IOPS bottleneck.
- Large file sequential throughput: On cache hit, Nydus+JuiceFS achieves GB/s sequential read throughput. Compared to traditional OCI, which is limited by single‑node local disk performance, JuiceFS can leverage NVMe distributed cache to provide high‑bandwidth data access.
Summary: combining on‑demand loading with shared cache
Nydus changes image loading from full pull to on‑demand reading, while JuiceFS places data blocks into a shared storage and distributed cache system. Together, they reduce container cold‑start time from 116s to 16s, and to ~1.4s with distributed cache warm-up; the data read volume during startup drops from 11 GB to 11 MB.
The value of this solution extends beyond just image startup acceleration. For AI inference scenarios, image blobs and model weights can be uniformly hosted by JuiceFS, sharing the same caching and prefetching mechanisms. During large‑scale scaling, distributed caching also reduces duplicate reads of the same data across many instances. This lessens concurrent pressure on the registry and backend storage.
In environments lacking shared storage, Dragonfly+Nydus is an excellent P2P alternative. However, in clusters that already have JuiceFS infrastructure, Nydus+JuiceFS delivers extreme image distribution and model loading acceleration with minimal operational overhead.
If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.
Mentions
juicefs finally fixed its severe and everpresent both transient and permanent data corruption issue, thats really nice juicefs#7555
How to Access JuiceFS via S3 and WebDAV Protocols Different applications access data in different ways: some rely on POSIX file interfaces, some are built on the S3 API, and some users want to browse and edit remote files through a file manager. JuiceFS' multi-protocol access capability is designed precisely to let these tools access the same file system. In this article, we'll explain the shared initialization path of S3 Gateway and WebDAV, describe their applicable scenarios and common mappings, and dive into their implementation principles and boundaries. A shared initialization path JuiceFS consists of a metadata engine and object storage: - The metadata engine manages directories, attributes, locks, and sessions. - Object storage stores the actual file data blocks. !Image description From an implementation perspective, S3 Gateway and WebDAV first go through the same service initialization path. The service process creates a JuiceFS client based on the metadata address, listen address, and common parameters such as subdir, read-only, cache, limits, access-log, and metrics. Then, it initializes the metadata client, object storage, data block cache, and VFS configuration in sequence, ultimately obtaining an fs.FileSystem. !Image description In other words, the common foundation of the two is not a FUSE mount point, but a native JuiceFS client with sessions, caching, metrics, and background tasks. Before S3 requests and WebDAV requests enter their respective protocol adaptation layers, they already share the same JuiceFS file system and the same file system namespace. Applicable scenarios and common mappings The shared initialization path explains how the two connect to the same file system. In actual use, what matters more is choosing the access method: which scenarios are suitable for S3 Gateway, which are suitable for WebDAV, and how common requests map to JuiceFS operations. - Linux applications, training jobs, or workloads requiring full file system semantics should prefer FUSE/CSI. - Applications already built on the S3 SDK, AWS CLI, MinIO Client, or object storage toolchains should prefer S3 Gateway. - Desktop file manager access, DAV/rclone synchronization, Office/apps collaboration, and simple sharing should prefer WebDAV. Mapping of common S3 Gateway APIs to JuiceFS file system operations: | S3 API | Purpose | Equivalent file system operation | | :---- | :---- | :---- | | GetObject | Download an object | open/read | | PutObject | Upload or overwrite an object | create/write/rename | | HeadObject | Query object metadata | stat/getxattr | | DeleteObject(s) | Delete one or more objects | unlink/remove | | ListObjects V1/V2 | List objects by prefix | readdir+filter | | CopyObject | Copy an object server-side | copy/read+write | | MakeBucket | Create a bucket | mkdir | | DeleteBucket | Delete an empty bucket | rmdir | | NewMultipartUpload | Create a multipart upload session | create | | PutObjectPart | Write a part | write | | CompleteMultipart | Merge and commit the object | copy + atomic rename | | Tags/Metadata | Read/write tags and metadata | get/setxattr | Mapping of common WebDAV methods to JuiceFS file system operations: | HTTP method | Purpose | Equivalent file system operation | | :---- | :---- | :---- | | GET | Download a file | read | | PUT | Upload or overwrite a file | write/create | | DELETE | Delete a file or directory | remove | | MKCOL | Create a directory | mkdir | | PROPFIND | Query properties or list a directory | stat/readdir | | PROPPATCH | Modify custom properties | setxattr | | COPY | Copy a file or directory | cp | | MOVE | Move or rename | mv/rename | | LOCK | Lock resources | File lock | | UNLOCK | Unlock resources | Release a file lock | | OPTIONS | Query server capabilities | Capability negotiation | | HEAD | Query HTTP metadata only | Lightweight stat | S3 Gateway principles: from object semantics to file system operations S3 Gateway converts S3 protocol requests into JuiceFS file system operations. This section covers four aspects: - The request path - Internal state - Key semantic conversions - Permission coordination Request path: from the MinIO S3 server to jfsObjects The MinIO S3 server handles the S3 protocol layer, including request routing, signature verification, authentication and authorization, and multipart upload orchestration. After requests pass through the protocol layer, they enter the ObjectLayer implemented by JuiceFS, namely jfsObjects. jfsObjects converts object-level reads, writes, listings, deletions, and attribute access into JuiceFS file system calls. !Image description State storage: how the internal namespace carries the Gateway state During request conversion, the Gateway does not only access user-visible object paths; it also uses an internal namespace to store state needed by the service itself. For example, .sys/tmp is used for temporary writes, .sys/uploads is used for multipart upload state, and .minio.sys is used to store MinIO/IAM-related metadata. These internal paths are not application data in the ordinary user's understanding, but they are the foundation the Gateway needs to maintain upload processes, part states, and permission configurations. Semantic conversion: PutObject, Multipart, Bucket, and Metadata The S3 object model and the file system model are not completely the same, so the Gateway needs to perform semantic conversions at several key points. PutObject must ensure that an object is visible only after the upload is complete. To prevent clients from reading incomplete objects, JuiceFS S3 Gateway writes to a temporary file and saves object attributes such as metadata and tag, commits to the target path via rename after completion. Leveraging the atomicity of file system rename, the Gateway can switch an object write from “uncommitted” to “committed.” !Image description Multipart upload needs to merge multiple parts into the final object. JuiceFS S3 Gateway can use file system capabilities such as CopyFileRange to combine already written parts and commit the final file via rename. This mainly adjusts file system metadata relationships rather than moving all data again. !Image description Bucket mapping handles the relationship between the object namespace and file paths. In single-bucket mode, the file system name corresponds to the bucket; in multi-bucket mode, top-level directories map to buckets. Object attributes such as metadata, tag, and ETag are stored through file extended attributes (xattr). Permission coordination: IAM and multiple instances S3 Gateway also needs to handle identity and permission information, including root users, regular users, policies, service accounts, and more. These configurations are stored in the JuiceFS file system through internal paths, allowing them to be reused across multiple Gateway instances. Therefore, multiple Gateway instances can share the same set of IAM configurations; however, each instance caches IAM information, so permission changes are not synchronized instantaneously. In actual deployments, we need to consider the refresh cycle. WebDAV principles: from file protocol to JuiceFS operations WebDAV starts from a different point than S3 Gateway. It targets file access scenarios, with common clients including macOS Finder, DAV/rclone, and some Office/apps clients. Since the WebDAV protocol includes concepts such as files, directories, and attributes, it does not need to first handle the model conversion between buckets, objects, and keys on the one hand and paths and files on the other, unlike S3 Gateway. Request path: from HTTP entry to webdavFS After a WebDAV request enters JuiceFS: 1.It passes through net/http and indexHandler. This layer mainly handles HTTP entry-related capabilities, such as authentication, transport, and conversion of directory access requests. 2.The request enters x/net/webdav.Handler, which handles the WebDAV protocol layer and converts protocol methods into file access requests that webdavFS can process. !Image description 3.After entering webdavFS, the file access requests passed in by the Handler are converted into JuiceFS file system calls such as open, create, read/write, rename, delete, and attribute setting. Since the WebDAV protocol is close to file system semantics, the focus here is not to rebuild an object model, but to handle how request methods, file open flags, and attribute states map onto JuiceFS operations. State boundaries: dead properties and LOCK WebDAV adaptation involves not only data reads and writes but also the protocol's own state. Dead properties can be saved as JSON in xattr, thus landing in JuiceFS' file attribute system; however, LOCK state is different—it's currently kept in the service process memory. Single-instance deployments usually have no problem; in multi-instance deployments, different instances cannot automatically synchronize lock state. These limitations usually do not affect Finder browsing, simple sharing, or DAV/rclone synchronization. However, if an application strongly relies on WebDAV file lock semantics, deployment mode and client behavior need to be verified separately. S3 Gateway and WebDAV share the same JuiceFS client initialization path and access the same file system namespace. The difference is that S3 Gateway needs to convert object semantics into file system operations, while WebDAV is closer to file access semantics. Therefore, the choice of access method should ultimately depend on the application itself: whether it organizes data as objects or operates on data as files and directories. If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.
Nydus + JuiceFS: Reducing Container Startup Time for AI Inference from 116s to 1.4s In large-scale AI inference services, when a cluster scales out, newly added inference instances must go through a series of cold-start steps before they can serve requests: container image preparation, file system mounting, runtime and inference framework initialization, and model weight loading. As image sizes and model weights continue to grow, the time spent on data preparation becomes increasingly prominent. During large-scale scaling events, many instances start simultaneously, further increasing the concurrent pressure on the registry, network, and backend storage. To address these issues, the industry has developed several optimization strategies, including reducing image size, using P2P distribution to alleviate concurrent pull pressure, and employing on-demand loading to reduce data transfer on the critical startup path. In this article, we’ll explore a shared‑storage‑based approach: combining Nydus’ on‑demand image loading capabilities with JuiceFS’ distributed storage and caching mechanisms. By sharing storage and reusing cached data, this approach reduces redundant data reads during large-scale scale-outs and shortens container startup time. For AI inference scenarios, container images and model weights can be managed together under a unified storage, caching, and prefetching system, accelerating both types of data while reducing the complexity of building and maintaining two separate data pipelines. Why OCI image distribution is slow Traditional Open Container Initiative (OCI) images are typically organized as compressed tar layers. In the image‑pull mode, a node must fully download and extract the required layers before the container can start. In practice, however, only about 6%–20% of files in an image are actually accessed during container startup; the rest of the download and extraction overhead is wasted on non‑critical‑path data. In large clusters, this “prepare everything first, then start” approach causes three problems: - Long cold‑start times: For GB‑sized images (like AI inference images), full download and extraction significantly delay new instance startup, often taking minutes. - Registry bottleneck: During HPA‑triggered bursts or large‑scale rollouts, hundreds of nodes simultaneously request GB‑sized layers from the registry, easily overwhelming the registry and congesting the network. - Redundant network and storage usage: Without an efficient P2P or distributed sharing mechanism between nodes, the same layers are repeatedly transferred across the cluster, resulting in significant bandwidth consumption and duplicated storage. Dragonfly+Nydus: what it solves and what it leaves behind To mitigate registry bandwidth pressure during concurrent image pulls in large clusters, a common solution is to introduce P2P distribution systems like Dragonfly. The core idea is to build a P2P distribution network via a scheduler and node‑side agents, allowing nodes to share image data and reduce egress traffic from the registry. However, P2P distribution primarily optimizes the data transfer path; it does not change the fundamental OCI mechanism of downloading and extracting layers before starting the container. Nydus, on the other hand, changes the loading approach: it transforms “prepare everything then start” into “fetch lightweight metadata first, and then read actual data on demand,” reducing data transfer on the critical startup path. Combined, Nydus handles on‑demand loading, while Dragonfly distributes the on‑demand data, reducing both the actual image read volume and the concurrent access pressure on the registry. However, this combination introduces additional infrastructure and operational complexity. Besides Nydus snapshotter, one must deploy and maintain Dragonfly’s scheduler, distributor, and node‑side components, and handle P2P network connectivity, cache space management, node anomalies, and cache eviction. In large‑scale GPU inference clusters, this means maintaining a separate data distribution and caching system independent of the storage layer. This leads to an alternative thought: if the cluster already has a mature distributed file system like JuiceFS, could image data distribution and caching be offloaded to the shared storage system, reusing existing storage infrastructure while reducing the cost of building and maintaining extra P2P components? Architecture redesign: Nydus blobs hosted by JuiceFS The core idea of the Nydus+JuiceFS solution is to transform image data distribution—originally relying on P2P network scheduling—into shared storage and cache reuse based on a distributed file system. In this architecture, JuiceFS serves as the underlying unified data engine hosting Nydus image blob data, and worker nodes read data through the standard POSIX interface. !Image description After an image is built and converted to Nydus format during the CI/CD phase, it’s split into two parts: - Bootstrap (metadata): Very small (typically a few hundred KB, for example, ~450KB), stored in the Image Registry, used for fast indexing. - Blobs (actual data blocks): Contains all file content (typically several GB, for example, ~1GB), persisted directly to the JuiceFS shared file system after conversion (for example, /var/lib/nydus/blobs). When a worker node starts a container, containerd triggers nydus-snapshotter, and nydusd mounts the container’s read‑only layer via FUSE. Then, when the application issues actual read requests, nydusd looks up the index in the Bootstrap and reads the corresponding blob data on‑demand through the JuiceFS mount point. The key change in this path is: the registry only distributes the manifest and the extremely lightweight bootstrap; the massive blob data transfer is entirely handled by JuiceFS’ distributed caching system. This fundamentally eliminates the registry network bottleneck during high‑concurrency pulls. For AI inference scenarios, this architecture can further unify the storage paths for images and model weights. In addition to Nydus image blobs, model weights can also be stored in JuiceFS, allowing both types of data to reuse the same storage, caching, and prefetching mechanisms. This accelerates data loading while reducing the complexity of building and maintaining two data pipelines. | Comparison basis | OCI full image pull | Dragonfly+Nydus | JuiceFS+Nydus | | :---- | :---- | :---- | :---- | | Core concept | Full layer pull + layer extraction | P2P network offloading + on‑demand image loading | Storage‑layer decoupling + distributed caching + on‑demand loading | | Cold start method & time | Full download and extraction required; typically 30s to 10min+ | Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds | Only download hundreds of kilobytes of bootstrap metadata and blobs needed for startup; usually seconds | | Image data transfer path | Registry → worker (full local download & extraction) | Registry / source → Dragonfly P2P network → worker on‑demand read | JuiceFS → worker on‑demand read via POSIX, accelerated by local & distributed caches | | Core components | Registry + containerd | Nydus, Manager, Scheduler, Seed Peer, and dfdaemon | JuiceFS + nydus‑snapshotter | | Image & model weight synergy | Image pulled from the registry to local node; model weights loaded via a separate path (NAS or object storage); two systems managed independently | Images loaded on‑demand via Nydus, distributed by Dragonfly; model weights still rely on independent external storage | Image blobs and model weights stored together in JuiceFS, sharing the same cache and warm-up mechanisms | | Operational complexity | Easy to overwhelm registry bandwidth under high concurrency | Requires extra maintenance of P2P scheduler, node network, and caching system | Can reuse existing JuiceFS storage & cache infrastructure, reducing independent image distribution components | | Local disk pressure | High: Stores full image and extracted data | High and unpredictable: Requires cache space for P2P; rapid scaling may cause disk full | Controllable: Unified shared storage pool, uses local NVMe for P2P distributed cache | | Network requirements | Standard TCP | Requires widely open P2P ports across nodes; may be invalid in cross‑VPC/NAT environments | Standard LAN – only requires normal high‑bandwidth inter‑node LAN; no intrusive network requirements | | Suitable scenarios | General workloads with no cold‑start requirements and small clusters (<20 nodes) | Ultra‑large clusters across data centers / clouds / edge computing environments without unified high‑bandwidth storage; strong ops capabilities of the team | Ultra-large clusters across data centers / clouds / edge computing environments without unified high-bandwidth storage, where JuiceFS is already deployed or can be deployed, aiming for unified image+model management and simplified ops | Runtime mechanism: how Nydus and JuiceFS coordinate Startup phase: fetching bootstrap (metadata) When the application starts, nydusd only needs to fetch the lightweight bootstrap file tree from the registry. The bootstrap describes the image file tree and block indexes, but contains no actual file content. After the container root filesystem (rootfs) is mounted, the subsequent startup process proceeds. In our test scenarios, containers can be ready and pass health checks within hundreds of milliseconds, without transferring the full large image data before startup. Read phase: on‑demand blob access When the application process actually issues read operations (such as open() or read()), nydusd captures the I/O requests via the Linux FUSE mechanism and locates the corresponding blob data block and offset using the bootstrap index. This process is completely transparent to upper‑layer applications. For the underlying data path, the read granularity changes from full image layers to the data blocks corresponding to specific files and offsets. nydusd directly accesses the JuiceFS mount point (for example, /var/lib/nydus/blobs) to read the target block, completely eliminating the runtime fetch‑from‑registry path. Cache phase: data reuse via JuiceFS cache groups JuiceFS’ distributed cache groups can reuse already‑cached blob data across the cluster. On a cache hit, data is returned to the kernel at near‑local‑NVMe speed. On a cache miss, only the specified offset block is pulled on‑demand and automatically filled into the cache. In addition to the automatic fill‑on‑miss, blobs can be pre-loaded into the cache to ensure full cache hits during on‑demand reads. For AI inference scenarios, the same mechanism can also be applied to loading model weight files. Production test data In production, we compared the solutions above against a large AI inference image. Container startup time comparison The table below shows the test results for container startup time: | Scenario | OCI overlayfs | Nydus+JuiceFS | Comment | | :---- | :---- | :---- | :---- | | Cold pull + run | 116 seconds | 16 seconds | OCI uses public cloud registry; Nydus+JuiceFS uses JuiceFS + public cloud object storage | | Cold pull + run (with distributed cache warm-up) | N/A | ~1.4 seconds | Required blobs pre‑loaded in JuiceFS distributed cache | | Warm pull + run | ~0 seconds | 0.22 seconds | OCI: same tag already on node; Nydus+JuiceFS: blobcache local cache | From the test results, we can see: - 7x speedup without cache warm-up: When neither local nor distributed cache is warmed up, Nydus+JuiceFS reduces container startup time from 116s to 16s. - ~99.9% reduction in data read volume during startup: In the cold pull + run scenario, read volume drops from 11 GB to 11 MB. Traditional OCI requires full download and extraction before startup, while Nydus+JuiceFS only fetches bootstrap metadata and reads blobs actually accessed during startup. - 1.4s startup with distributed cache warm-up: When blobs are cached in JuiceFS distributed cache, startup takes 1.4s; with local blobcache warmed up (warm pull + run), startup takes only 0.22s. On‑demand file read performance The table below shows test results for: on-demand file read performance: | Scenario | Traditional OCI (local disk direct read) | Nydus+JuiceFS (distributed cache hit) | | :---- | :---- | :---- | | First‑block read latency (TTFB) | ~0.9 milliseconds | ~3.2 milliseconds | | Small file / scattered configuration loading (IOPS) | Baseline | Close to baseline | | Large file sequential read throughput | Local disk limit | GB/s level (JuiceFS NVMe distributed cache) | From the test results, we can see: - Time to first byte (TTFB): On distributed cache hit, first‑block read latency is ~3.2 milliseconds, about 2.3 milliseconds higher than local disk direct read (0.9 milliseconds). The extra latency comes from the FUSE data path and cache access, but remains at millisecond level, generally imperceptible to the application side. - Small file IOPS: Performance is very close to local disk baseline. In scenarios with dense Python dependency and configuration file reads, JuiceFS caching introduces no noticeable IOPS bottleneck. - Large file sequential throughput: On cache hit, Nydus+JuiceFS achieves GB/s sequential read throughput. Compared to traditional OCI, which is limited by single‑node local disk performance, JuiceFS can leverage NVMe distributed cache to provide high‑bandwidth data access. Summary: combining on‑demand loading with shared cache Nydus changes image loading from full pull to on‑demand reading, while JuiceFS places data blocks into a shared storage and distributed cache system. Together, they reduce container cold‑start time from 116s to 16s, and to ~1.4s with distributed cache warm-up; the data read volume during startup drops from 11 GB to 11 MB. The value of this solution extends beyond just image startup acceleration. For AI inference scenarios, image blobs and model weights can be uniformly hosted by JuiceFS, sharing the same caching and prefetching mechanisms. During large‑scale scaling, distributed caching also reduces duplicate reads of the same data across many instances. This lessens concurrent pressure on the registry and backend storage. In environments lacking shared storage, Dragonfly+Nydus is an excellent P2P alternative. However, in clusters that already have JuiceFS infrastructure, Nydus+JuiceFS delivers extreme image distribution and model loading acceleration with minimal operational overhead. If you have any feedback on this article or ideas to share, we invite you to participate in the discussions on GitHub and join our community on Discord.