CernVM-FS Swissknife Utility

The CernVM-FS Swissknife utility provides a set of commands which are used internally. These commands may be useful for custom low-level programmatic management of CVMFS repositories.

cvmfs_swissknife utility is always compiled and installed during a build from source. It is packaged for Red Hat based and Debian based platforms in the cvmfs-server package.

ingestsql

This is a utility for changing contents of a CVMFS repository. It is not used by cvmfs_server publish but lets you achieve the same outcome.

cvmfs_swissknife ingestsql tells CVMFS Gateway to update the catalogs according to what is in the SQLite database file supplied as an input parameter.

In the database, there are descriptions of files, directories and links to create or modify, and the list of entities to delete. These entries contain metadata such as filename, permissions mode, modification time, owner user and group number, size, hash etc.

ingestsql connects to CVMFS Gateway and relays such metadata updates to it. Gateway then incorporates these info the catalogs and increments the revision, making the updates visible to all clients.

For these metadata updates to be correct, the newly added files listed in the database must be placed at the locations specified, using other tools. For S3-based CVMFS repository, s3cmd or aws s3 cp commands can be used. See also cvmfs-posix-tools, where utilities such as cvmfs-rsync combine the functionality of uploading the actual files and updating the repository metadata using ingestsql.

Command line arguments

 # cvmfs_swissknife help
...

Command ingestsql
--
Graft the contents of a SQLite DB to the repository
Options:
  -D    input sqlite DB
  -N    fully qualified repository name
  -g    gateway URL (optional)
  -w    stratum 0 base url (optional)
  -t    temporary directory (will try TMPDIR if not set) (optional)
  -@    proxy URL (optional)
  -k    public key (optional)
  -l    lease path (optional)
  -p    prefix to add to lease and all graft files (optional)
  -q    number of concurrent write jobs (optional)
  -s    gateway secret (optional)
  -3    s3 config (optional)
  -a    Allow additions (default true, false if -d specified) (optional)
  -d    Allow deletions (optional)
  -x    Force deletion of any lease (optional)
  -c    Enable corefile generation (requires ulimit -c >0) (optional)
  -n    create empty database file (optional)
  -C    config prefix, default /etc/cvmfs/gateway-client/ (optional)
  -B    mount point to block on pending visibility of update (optional)
  -W    Timeout, in seconds, for waiting on pending visibility of update (default: if -B given, wait infinitely; if no -B, immediate exit) (optional)
  -T    reset TTL in sec (optional)
  -z    Create missing nested catalogs (optional)
  -r    lease retry interval (optional)
  -Z    check and set completed_graft property (optional)
  -P    priority for graft (integer) (optional)
  -v    Enable verbose logging (optional)

Configuration files

cvmfs_swissknife ingestsql uses an optional config file at a location <config-prefix>/<repo name>/config, where config-prefix is set by -C command line argument, defaulting to /etc/cvmfs/gateway-client/.

The config file can contain the following parameters. Each parameter has a corresponding command line option which overrides the config file.

CVMFS_GATEWAY - HTTP URL for requests to CVMFS Gateway (-g);
CVMFS_STRATUM0 - HTTP URL for repo access (-w);
CVMFS_HTTP_PROXY - HTTP proxy URL, or literal DIRECT (-@).

Examples

cvmfs_swissknife ingestsql -D '' -N '' -n my.db
echo HELLO > hello_ingestsql.txt
wc --bytes hello_ingestsql.txt # 6
sha1sum hello_ingestsql.txt # a8eec30a5b2d71bc890175f5b361ebb28d7c54a8
sqlite3 my.db <<< "insert into files (name, size, hashes) values ('hello_ingestsql.txt', 6, 'a8eec30a5b2d71bc890175f5b361ebb28d7c54a8');"
s3cmd put hello_ingestsql.txt s3://mybucket/my.repo.name/external/hello_ingestsql.txt
cvmfs_swissknife ingestsql \
    -D my.db \
    -N my.repo.name \
    -g http://cvmfs-gateway-server/api/v1 \
    -w http://s3-server/mybucket/my.repo.name \
    -@ http://proxy-server \
    -k /etc/cvmfs/keys/my.repo.name.pub \
    -s /etc/cvmfs/keys/my.repo.name.gw \
    -3 /etc/cvmfs/s3.conf \
    -t /tmp

Tables structure

cvmfs_swissknife ingestsql provides a special mode of invocation which initializes an empty database file with correct schema: cvmfs_swissknife ingestsql -D '' -N '' -n my.db.

Here is the schema and the contents of the database after the creation by the above command:

CREATE TABLE IF NOT EXISTS dirs (
    name  TEXT    PRIMARY KEY,
    mode  INTEGER NOT NULL DEFAULT 493, -- 0755 in octal
    mtime INTEGER NOT NULL DEFAULT (unixepoch()),
    owner INTEGER NOT NULL DEFAULT 0,
    grp   INTEGER NOT NULL DEFAULT 0,
    acl   TEXT    NOT NULL DEFAULT "",
    nested INTEGER DEFAULT 1
);
CREATE TABLE IF NOT EXISTS files (
    name   TEXT    PRIMARY KEY,
    mode   INTEGER NOT NULL DEFAULT 420, -- 0644 in octal
    mtime  INTEGER NOT NULL DEFAULT (unixepoch()),
    owner  INTEGER NOT NULL DEFAULT 0,
    grp    INTEGER NOT NULL DEFAULT 0,
    size   INTEGER NOT NULL DEFAULT 0,
    hashes TEXT    NOT NULL DEFAULT "",
    internal INTEGER NOT NULL DEFAULT 0,
    compressed INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS links (
    name   TEXT    PRIMARY KEY,
    target TEXT    NOT NULL DEFAULT "",
    mtime  INTEGER NOT NULL DEFAULT (unixepoch()),
    owner  INTEGER NOT NULL DEFAULT 0,
    grp    INTEGER NOT NULL DEFAULT 0,
    skip_if_file_or_dir INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS deletions (
    name      TEXT PRIMARY KEY,
    directory INTEGER NOT NULL DEFAULT 0,
    file      INTEGER NOT NULL DEFAULT 0,
    link      INTEGER NOT NULL DEFAULT 0
);
-- Table used by CVMFS to store different properties such as schema version and revision.
CREATE TABLE IF NOT EXISTS properties (
    key   TEXT PRIMARY KEY,
    value TEXT NOT NULL
);
INSERT INTO properties VALUES
    ("schema_revision", "4")
;