.. _cpt_swissknife:

.. highlight:: none

CernVM-FS Swissknife Utility
============================

The CernVM-FS Swissknife utility provides a set of commands which are used internally.
These commands may be useful for custom low-level programmatic management of CVMFS repositories.

``cvmfs_swissknife`` utility is always compiled and installed during a build from source.
It is packaged for Red Hat based and Debian based platforms in the ``cvmfs-server`` package.


ingestsql
---------

This is a utility for changing contents of a CVMFS repository.
It is not used by ``cvmfs_server publish`` but lets you achieve the same outcome.

``cvmfs_swissknife ingestsql`` tells CVMFS Gateway to update the catalogs according to what is in the SQLite database file supplied as an input parameter.

In the database, there are descriptions of files, directories and links to create or modify, and the list of entities to delete.
These entries contain metadata such as filename, permissions mode, modification time, owner user and group number, size, hash etc.

``ingestsql`` connects to CVMFS Gateway and relays such metadata updates to it.
Gateway then incorporates these info the catalogs and increments the revision, making the updates visible to all clients.

For these metadata updates to be correct, the newly added files listed in the database must be placed at the locations specified, using other tools.
For S3-based CVMFS repository, ``s3cmd`` or ``aws s3 cp`` commands can be used.
See also ``cvmfs-posix-tools``, where utilities such as ``cvmfs-rsync`` combine the functionality of uploading the actual files and updating the repository metadata using ``ingestsql``.

Command line arguments
~~~~~~~~~~~~~~~~~~~~~~

::

     # cvmfs_swissknife help
    ...

    Command ingestsql
    --
    Graft the contents of a SQLite DB to the repository
    Options:
      -D    input sqlite DB
      -N    fully qualified repository name
      -g    gateway URL (optional)
      -w    stratum 0 base url (optional)
      -t    temporary directory (will try TMPDIR if not set) (optional)
      -@    proxy URL (optional)
      -k    public key (optional)
      -l    lease path (optional)
      -p    prefix to add to lease and all graft files (optional)
      -q    number of concurrent write jobs (optional)
      -s    gateway secret (optional)
      -3    s3 config (optional)
      -a    Allow additions (default true, false if -d specified) (optional)
      -d    Allow deletions (optional)
      -x    Force deletion of any lease (optional)
      -c    Enable corefile generation (requires ulimit -c >0) (optional)
      -n    create empty database file (optional)
      -C    config prefix, default /etc/cvmfs/gateway-client/ (optional)
      -B    mount point to block on pending visibility of update (optional)
      -W    Timeout, in seconds, for waiting on pending visibility of update (default: if -B given, wait infinitely; if no -B, immediate exit) (optional)
      -T    reset TTL in sec (optional)
      -z    Create missing nested catalogs (optional)
      -r    lease retry interval (optional)
      -Z    check and set completed_graft property (optional)
      -P    priority for graft (integer) (optional)
      -v    Enable verbose logging (optional)

Configuration files
~~~~~~~~~~~~~~~~~~~

``cvmfs_swissknife ingestsql`` uses an optional config file at a location ``<config-prefix>/<repo name>/config``, where ``config-prefix`` is set by ``-C`` command line argument, defaulting to ``/etc/cvmfs/gateway-client/``.

The config file can contain the following parameters.
Each parameter has a corresponding command line option which overrides the config file.

* ``CVMFS_GATEWAY`` - HTTP URL for requests to CVMFS Gateway (``-g``);
* ``CVMFS_STRATUM0`` - HTTP URL for repo access (``-w``);
* ``CVMFS_HTTP_PROXY`` - HTTP proxy URL, or literal ``DIRECT`` (``-@``).

Examples
~~~~~~~~

.. code:: bash

    cvmfs_swissknife ingestsql -D '' -N '' -n my.db
    echo HELLO > hello_ingestsql.txt
    wc --bytes hello_ingestsql.txt # 6
    sha1sum hello_ingestsql.txt # a8eec30a5b2d71bc890175f5b361ebb28d7c54a8
    sqlite3 my.db <<< "insert into files (name, size, hashes) values ('hello_ingestsql.txt', 6, 'a8eec30a5b2d71bc890175f5b361ebb28d7c54a8');"
    s3cmd put hello_ingestsql.txt s3://mybucket/my.repo.name/external/hello_ingestsql.txt
    cvmfs_swissknife ingestsql \
        -D my.db \
        -N my.repo.name \
        -g http://cvmfs-gateway-server/api/v1 \
        -w http://s3-server/mybucket/my.repo.name \
        -@ http://proxy-server \
        -k /etc/cvmfs/keys/my.repo.name.pub \
        -s /etc/cvmfs/keys/my.repo.name.gw \
        -3 /etc/cvmfs/s3.conf \
        -t /tmp

Tables structure
~~~~~~~~~~~~~~~~

``cvmfs_swissknife ingestsql`` provides a special mode of invocation which initializes an empty database file with correct schema: ``cvmfs_swissknife ingestsql -D '' -N '' -n my.db``.

Here is the schema and the contents of the database after the creation by the above command:

.. code:: sql

    CREATE TABLE IF NOT EXISTS dirs (
        name  TEXT    PRIMARY KEY,
        mode  INTEGER NOT NULL DEFAULT 493, -- 0755 in octal
        mtime INTEGER NOT NULL DEFAULT (unixepoch()),
        owner INTEGER NOT NULL DEFAULT 0,
        grp   INTEGER NOT NULL DEFAULT 0,
        acl   TEXT    NOT NULL DEFAULT "",
        nested INTEGER DEFAULT 1
    );
    CREATE TABLE IF NOT EXISTS files (
        name   TEXT    PRIMARY KEY,
        mode   INTEGER NOT NULL DEFAULT 420, -- 0644 in octal
        mtime  INTEGER NOT NULL DEFAULT (unixepoch()),
        owner  INTEGER NOT NULL DEFAULT 0,
        grp    INTEGER NOT NULL DEFAULT 0,
        size   INTEGER NOT NULL DEFAULT 0,
        hashes TEXT    NOT NULL DEFAULT "",
        internal INTEGER NOT NULL DEFAULT 0,
        compressed INTEGER NOT NULL DEFAULT 0
    );
    CREATE TABLE IF NOT EXISTS links (
        name   TEXT    PRIMARY KEY,
        target TEXT    NOT NULL DEFAULT "",
        mtime  INTEGER NOT NULL DEFAULT (unixepoch()),
        owner  INTEGER NOT NULL DEFAULT 0,
        grp    INTEGER NOT NULL DEFAULT 0,
        skip_if_file_or_dir INTEGER NOT NULL DEFAULT 0
    );
    CREATE TABLE IF NOT EXISTS deletions (
        name      TEXT PRIMARY KEY,
        directory INTEGER NOT NULL DEFAULT 0,
        file      INTEGER NOT NULL DEFAULT 0,
        link      INTEGER NOT NULL DEFAULT 0
    );
    -- Table used by CVMFS to store different properties such as schema version and revision.
    CREATE TABLE IF NOT EXISTS properties (
        key   TEXT PRIMARY KEY,
        value TEXT NOT NULL
    );
    INSERT INTO properties VALUES
        ("schema_revision", "4")
    ;