fedops blog

Privacy in Computing

Mon 06 March 2023

My FOSS Photo Workflow

Posted by fedops in Software   

Digital photography workflows tend to be dominated by a relatively small number of proprietary, commercial software suites that have been optimized over years or even decades. Adobe Lightroom, Capture One, or Affinity Photo are household names, and they're all available for both MacOS as well as Windows. Which also means that most prosumer and professional photographers use those operating systems as the fine-tuned applications are their bread-and-butter tools they generally can't do without.

A small number of open source enthusiasts develop and use some remarkable, but largely unremarked, programs that aim to replace those walled gardens and also work on free operating systems such as Linux. A very good site to find information is pixls.us.

In the following paragraphs I'll outline my own personal workflow which might provide some useful pointers if you're just starting out. I'm shooting using a Canon DSLR and use Fedora Linux, but I believe most things should work regardless of camera manufacturer and distro choice.

Copying Images off the Camera

The first step in the workflow is getting the pixels out of the camera. One option is to pop the memory card into a card reader and transfer them, e.g. using rapid photo downloader.

I don't have a Compact Flash reader so I prefer to connect the camera to my PC with a USB cable. I then use gphoto2 to obtain the data, which is the swiss army-knife of cli camera utilities. It is also wonderfully scriptable. Following are a few useful commands:

# show automatically detected cameras
gphoto2 --auto-detect
# summary info
gphoto2 --summary
# show number of files
gphoto2 -n
gphoto2 -n -f /store_00010001/DCIM/100EOS7D
# list folders
gphoto2 -l
# list all files
gphoto2 -L
# copy file range (by numbers)
gphoto2 --get-file 7-12
# only copy files newer than what's in the current dir
gphoto2 --new

Generally I use the gphotofs package to mount the camera's filesystem directly:

# mount camera
gphotofs /mnt/eos
# umount
fusermount -u /mnt/eos

This makes it possible to manipulate the in-camera image files using the usual commands such as cp and ls, or indeed using a file manager. Some examples:

# copy files not older than 7 days
cp --preserve=all `find /mnt/eos/store_00010001/DCIM/100EOS7D/ -mtime -7 -print` .
# copy files created between 2021-05-08 and 2021-05-10
cp --preserve=all `find /mnt/eos/store_00010001/DCIM/100EOS7D/ -type f \
    -newermt 2021-05-08 ! -newermt 2021-05-10 -print` .
# sync all files
rsync -av /mnt/eos/store_00010001/DCIM/100EOS7D/ .

These commands are quite useful. The rsync command should only be used if you keep all your image files in one directory, which is not a great way if you shoot a lot. This brings me to the next chapter and a very fundamental decision:

Photo Organization

I recommend to define a scheme of how you manage your image files, and then stick to it as strictly as possible. Here's mine:

  • I have one directory which holds all my images, and which I always synchronize and backup in its entirety.
  • for each trip/outing I create a directory inside of this directory, giving it a descriptive name. My format is something like spain-2023-01 or nature-2023-02. If I have a trip log or some other documentation of the shoot such as model release forms those will also be stored here. I call this the "shoot directory".
  • for extended trips I will create further subdirectories, usually one per day, such as day01, day02, ... This is especially necessary if there's a danger that the camera-internal image numbering will wrap around and thus image names may not be unique anymore. Each directory holds the RAW files from the camera from that day.
  • in the shoot directory there is a edits subdirectory into which I store all the edited files such as JPG exports.
  • I leave my file names as they come out of the camera, but you may chose to rename them to whatever scheme you would like. Rapid photo downloader and most other programs support on-the-fly bulk renaming.

Whatever scheme you use, make sure everything's settled in before proceeding to the next step, which is:

Culling the Herd

If you're like me, you will not want to keep all the images you brought back from a trip. In sport or wildlife photography for example usually up to 90% of the images are somehow defective and need to be deleted to maintain an overview of what's there and identify the "keepers".

My preferred way is to import the entire shoot directory into digikam using Import --> Add Images.... Digikam will then start to create thumbnails, it's best to give it some time to finish this process after importing. The user interface is somewhat unusual and definitely doesn't have the refined polish of Lightroom, but invest some time to familiarize yourself and it is suprisingly capable.

Next, go through all the images one by one, compare them, and delete undesirable ones. The easiest way is to mark images as "rejected" (red flag, which I have bound to the "x" key) and then later delete them all at once. The main objective of this step is to get rid of the cruft that won't be worked on anyway.

Now is also a good time to assign ratings (using the number keys 0-5) to make it easier to find the images I will process in the next step. Those images (I call them "keepers") get a rating of 3 stars. I usually don't rate others. My camera supports rating via a special button, which I will sometimes use when I got a shot I can already see is great even on the camera display. Digikam picks this rating up from the EXIF tag and you'll see it in this step.

Once I have gone through all images of the shoot I filter for "rejected" only, mark them all, and then delete them.

Assigning Metadata

Ensuring proper metadata makes filtering and later retrieval easier. I only orry about metadata in my "3 stars or better" images so the first step is to turn on the filter to see just them.

I have a fairly elaborate tree structure of tags, of which I assign the suitable ones in this step. For example I like to photograph airplanes, so a typical tag structure might look like:

  • Boeing
    • B737
    • B777
      • B777-200
      • B777-300ER
  • Airbus
    • A340

Assigning "B777-200" will also assign the "B777" and "Boeing" tags, which makes it easy to filter and search for those more generic terms later on.

The most time-consuming steps are to set the titles and captions. In my world the title describes the location and date of the image. So for example "Kruger Park, April 2020". Chances are it's the same for many or all of the images, so I select all required images and set the title at once.

The caption contains a description of the image, maybe "Lilac-breasted roller (Coracias caudatus) in a tree". Setting these is a bit of work, but this is searchable data so will come in quite handily later on when you need to find all pictures of rollers in your collection.

My camera has a GPS module so the geolocation data will be inside the RAW images. If you apply this data manually you may want to do it at this time.

This completes the library step if you're used to Lightroom. It is important to note that every image which should be processed later must be "touched" at this time. Modifying metadata means a sidecar file is created, which we need in the next step.

Editing

I do my editing in ART, so called because it is A fork of RawTherapee. There are many significant differences, such as a reordering of the tools to form a different and somewhat simplified workflow within the application. It also understands XMP sidecar files, which RawTherapee unfortunately does not, and which are key to this workflow. ART is available in multiple formats, including universally useful Appimages, which I use.

Start ART and load the directory to be worked on. This adds all the images to the current view. I then set the view filter to only show images rated 3 stars or more.

I do not edit my images heavily. In general my steps are:

  1. enter edit mode and load "resize1920x1600" profile for presets
  2. rotate to straighten lines, optionally correct perspective
  3. adjust exposure and contrast (lightly, if required)
  4. adjust highlights and shadows
  5. turn on post-resize sharpening
  6. update metadata (min. Description and Title, should be auto-filled from sidecar file)
  7. add to queue (Ctrl+b)

The "resize1920x1600" profile sets the output dimensions accordingly and turns on output sharpening. It needs to be defined once in the profile settings.

A nice quality of life improvement is that Control+C and Control+V will copy and paste the editing steps from one file to the next. So if you have multiple consecutive images they can be modified quickly and consistently.

Sometimes during editing I will discover that an image isn't as good as I thought. I will then down-rate it to 2 or even 1, which removes it from the current filtered view. If I need to find a better alternative from a set of similar images I disable the filter, find a substitute, rate it as 3 stars, and re-enable the filter.

Another nice feature are the color labels. Generally I label pictures as "green" when I am done working on them, and ones I need to revisit later as "red". This comes in very handy during lengthy editing sessions. If I break for the night and come back the next day I can see what has already been done and what I still need to work on.

Once I'm happy with the edits I run the queue which I have configured to export the files as 1920x1600 TIFFs into a subdirectory "edits" in the current directory (which will be created if required). If you have a fast machine you can set the queue to "auto run" which means any new entries will get processed in the background. On slower machines it might be better to run the queue when you don't need the horsepower for editing.

Postprocessing and Resizing

Two things remain to be done. I want to convert the TIFFs to JPGs with an overlaid watermark, and I want to pull in the metadata that was added so laboriously in digikam to the JPG. This is handled by two shell scripts working together. These require the exiftools and ImageMagick packages to be installed on your system.

First, I cd into the "edits" directory where ART dumped the TIFFs. Then this script will loop over all TIFFS and determine whether a JPG exists and isn't newer than the parent TIFF. It will then run the second script add-wm.sh on that file:

#!/bin/bash
#
for i in *.tif
do  base=`echo $i | sed 's/.tif//'`
    if [ "${base}-wm.jpg" -ot $i ]
    then    add-wm.sh $i
    fi
done

This second script is a bit more involved:

#!/bin/bash
# add watermark to image and convert to JPG

# "${1%.*}" is just the part of the filename up to the first '.'
echo -n ${1} + wm --\> "${1%.*}"-wm.jpg

# overlay watermark image and save as JPG
#composite -quiet -dissolve 50% -gravity southeast ~/.config/misc/watermark2020.png ${1} "${1%.*}"-wm.jpg
composite -quiet -dissolve 50% -gravity southeast ~/.config/misc/1x1.png ${1} "${1%.*}"-wm.jpg

# copy exiftags from TIF into newly created JPG
exiftool -tagsFromFile ${1} "${1%.*}"-wm.jpg >/dev/null 2>&1

# pull in tags from XMP sidecar file if it exists
# (if this TIF was generated by Lightroom it's missing those tags)
if [ -e "../${1%.*}".CR2.xmp ]
then    echo -n " ...with tags from ../${1%.*}".CR2.xmp
        exiftool -tagsFromFile "../${1%.*}".CR2.xmp -all:all "${1%.*}"-wm.jpg >/dev/null 2>&1
fi
echo ""

# copy Caption-Abstract to Description field (workaround for ART)
exiftool '-Description<Caption-Abstract' "${1%.*}"-wm.jpg >/dev/null 2>&1

# get rid of the "_original" file
rm -f "${1%.*}"-wm.jpg_original

Some explanations:

  • the two "composite" lines either add a watermark with my full name and logo or just a transparent pixel and create the JPG file. I comment out what I don't want.
  • the first exiftool command then transfers the tags from the TIFF to the JPG.
  • the if-block looks for an XMP sidecar file and if found will also add those tags to the JPG file.
  • the final exiftool command copies the Caption/Abstract field (which ART populates with the caption) to the IPCC Description field (which most software expects the caption to live in).

At this point I have the JPGs and will look through them to see what they look like. Sometimes a bit more editing may be required which just repeats the above steps in ART and then a rerun of the watermarks script rebuilds the JPGs of the changed files only.

Alternatives

This is just my somewhat specialized workflow. I'd be the first to agree it is quite involved but it works for me and doesnt shoehorn me into any proprietary software or huge binary image databases. Everything remains in individual files and there is no questionable exporting required should I decide to change software.

I have previously experimented with other software. One which I would be remiss in not mentioning is Darktable. It is an extremely capable all-in-one database management and editing application closely patterned after Lightroom. Where Lightroom is limited by Adobe's protectionism towards their flagship (and much more expensive) Photoshop suite, Darktable's authors pull out all the stops and thus the program has far surpassed the original's functionality.

Personally I was somewhat overwhelmed by it and found I got better results with ART but I still suggest Darktable is worth a try. If you work with photos regularly, possibly daily, it might be just the ticket. YMMV.