MAR (Multi-format acrhive)

mar is an archival tool with a file-type of the same name. It is written entirely in BASH and aims to find the optimal compression algorithm space-wise for each file.

Support for compression algorithms

Although the tool is portable, it does rely on the availability of utilities like xz, bzip2, gzip, brotli, zip and zstd. By default, all compression utilities available on the system are used to determine the one producing the smallest file. Users can also specify which compression utilities to use for the testing and archival process. Note that none of the compression utilities mentioned are included with mar, and decompression of files previously compressed with an utility not present on the system will result in mar producing a warning and skipping these.

Addition of new algorithms

New compression utilities can easily be added using a simple function. mar provides the ability to add new algorithms by sourcing the file ~/.config/mar/user.conf

A function named add_algorithm is defined within mar to provide an extremely flexible way to add new algorithms. Here is an explanation of the syntax:

add_algorithm	name	file-type	min-quality	max-quality	compression command	decompression command	function call
	The name of the utility as referenced in mar	Either * for all files, or comma-separated file-types (like *.zip,*.rar)	The minimum compression quality of the utility	The maximum quality	The command to compress a file	The command to decompress a file	(Optional) Functions to call after compression and decompression. Comma-separated, e.g. function-1,function-2

Setting min-quality and max-quality to the same value will always enforce the same value. Not using the %quality% placeholder described below effectively does the same.

The function has several placeholder variables:

Placeholder	Desscription
%in-file%	The input file to be compressed.
%out-file%	The output file captured by mar to be included into the archive.
%quality%	The compression quality provided by mar.

Aside from adding new general compression utilities, this can also be used for lossy compression. Here's a fictious convert-to-jpeg utility, defined using add_algorithm:

function jpeg_change_file_name
{
  # This function is called with a global variable present named $CURRENT_FILE.
  # This refers to the already compressed file.

  # There is also a variable named "$CURRENT_MODE" which is either "compress" or "decompress"
  # In this case, we only want to do anything of note during "compress" mode.
  if [ "$CURRENT_MODE" = "compress" ];
  then
    # Change file extension to .jpg
    CURRENT_FILE="${CURRENT_FILE%.*}.jpg  
  fi
}

# Setting a minimum quality of 5 so it will never look too horribe.
# No decompression necessary, so we simply make a copy.
add_algorithm "jpeg" "*.bmp,*.png,*.tga,*.raw" 5 9 "convert-to-jpg -q %quality% %in-file% %out-file%" "cp %in-file% %out-file%" jpeg_change_filename

This makes mar quite flexible in archiving tasks, being both useful for lossless archival as well as lossy archival. You could also, for instance, archive videos in a similar manner into more efficient formats. Other compression algorithms remain available as well of course, so if for whatever reason lossless compression beats out lossy compression, it is selected instead.

Other uses of add_algorithm

While mar creates archives first-and-foremost, the flexibility of the function allows for a number of things, just to name a few:

• Video/music re-encoding.
• Conversion from one document type to another (e.g. .docx to .pdf).
• Conversion from one lossless format to another smaller one (e.g. .bmp to .png).
• Replacement of a file to a web-hosted link for said file. While these ways of using mar will not produce an archive with data 1:1 to its input, it may be useful in some workflows.

Testing for the most suitable algorithm

Before deciding what method/algorithm to use to compress the file, several are tested in parallel to determine which one produces the smallest file. For files larger than 64MB, 64 1MB blocks are read from this file to produce a smaller file for testing. These blocks are selected evenly from the entirity of the file. So, for example, for a 128MB file, every other 1MB block is concatenated together to provide the test file. After finding the most efficient algorithm, a header and the contents of the compressed files are appened to the output archive.

Additional features

mar has a few unique features that make it user-friendly. Here's a few:

• When creating an archive, the user can specify a quality level from 0-100. This quality level is mapped to each compression utility's own quality level.
• Archives can be merged together simply by concatenating them together.

[unfinished]