Multi-Petabyte Data Distribution in Industry & Science with CernVM File System

CVMFS

Lazy pulling in action

Normal image pulling

cvmfs-posix-tools

cvmfs_chgrp
cvmfs_chmod
cvmfs_chown
cvmfs_insert
cvmfs_ln
cvmfs_mkdir
cvmfs_rm
cvmfs_rmdir
cvmfs_rsync
cvmfs_setfacl
cvmfs_touch

Publish command in action

<aside class="notes">
						
							Importantly, in this way, two expensive steps must occur to accomplish the task of publishing a new file.
						
							First, the new file is copied into local scratch area.
						
							SKIPPABLE BEGIN
							This is dealt by overlayfs, it provides a nice illusion of editing the filesystem directly, and it enables the next step, the publish command, to instantly see what are the changed files to upload.
							SKIPPABLE END
						
							Second, the copy in the scratch area is transferred to the primary storage of CVMFS repo.
						
							It can be a local directory if we are working directly on the main server, called Stratum 0.
						
							It can be S3 location.
						
							And it can be a remote CVMFS Gateway service.
						
						</aside>

<table width="100%">
							<tr width="100%">
								<td width="75%">
									<code style="white-space: nowrap;">cvmfs_server transaction</code>
								</td>
								<td width="25%">
									change overlayfs mode
								</td>
							</tr>
							<tr width="100%">
								<td width="75%">
									<code style="white-space: nowrap;">cp bigmodel.gguf /cvmfs/my.repo</code>
								</td>
								<td width="25%">
									copy to scratch area
								</td>
							</tr>
							<tr width="100%">
								<td width="60%">
									<code style="white-space: nowrap;">cvmfs_server publish</code>
								</td>
								<td width="25%">
									copy to primary storage
								</td>
							</tr>
						</table>

cvmfs_rsync in action

Q & A

Georgios Christodoulis, CERN georgios.christodoulis@cern.ch	Andriy Utkin hello@autkin.net

Open Source at CERN