I wonder if my hive crew have been missing my incessantly nonsensical comments or if they're enjoying the serenity.
As I mentioned at the end of the last progblog I wrote two weeks ago (x_x), I finally managed to pick up a cute laptop (I'm so used to getting the highest spec'd things that getting something practically off the shelf and only upgrading storage feels so weird and wrong) and get back into the overpriced (I'm now broke for the next 6 years) and very...um..."special" Apple ecosystem that I have this love/hate relationship with (love it because it's the one that gives me the least problems and I find their devices fun to use, and hate it because it is so very overpriced and some things make me -_-).
I've since done the major setup stuff (including using my wallpaper folder so I have less generic backgrounds which are more for the lockscreen than anything else as I rarely see the desktop) and it's just little tweaks and at some stage setting up my tablet.
The reasons for getting the small rig extended past my usual needing something mobile for the time periods where I'm going to be away from the big rig. The small rig is now also going to be housing the photo library partly as it's easier to drag around to show photos to relatives, and mostly because I've been having a lot of problems with importing photos on the big rig. Initially I thought that the photos weren't importing and that I'd lost a batch of them, which resulted in me not deleting anything off my phone just in case. Eventually I worked out that some of them were going into the wrong date folders but couldn't work out what I was doing wrong even after fiddling with settings. As both shotwell and digikam were doing the same things I decided there was probably something interesting going on with gphoto which both use as a backend and continued hoarding images on my phone as I had no idea where the missing photos were ending up.
My initial thoughts were to just import everything into Photos and sync the originals folder, which once upon a time used to just be date based like everything else.
No longer. Now it looks like
and has probably been like that for quite a while but I haven't used a Mac since my 2011 iMac. I could have synced that but I would have never been able to find anything.
So I grabbed a selection of files and started running some tests trying to work out how the hell I was going to make what I thought would be a relatively straightforward job work.
Pro-tip: nothing is ever straightforward when you mix operating systems. Different versions of the same operating systems also count as different operating systems.
After much stuffing around, I eventually decided that the easiest thing to do would be to import everything into Photos, set up a smart album to collect the latest imports (because currently the Imports view seems to show everything that was ever imported separated by the dates they were imported which requires me being more careful about clicking than I want to be) and export everything in that album to a sorting folder (not to be confused with a sorting hat) which will then trigger an Automator folder action and sort them nicely into my cloud sync folder in the date based structure I wanted and have been using for years (since 13th of August 2004 to be precise).
A few hours playing with Automator later and I realised I was going to need a shell script.
Do you think I could find a shell script that did what I wanted?
I spent a week scouring the internet and learning just enough zsh scripting to be a total and complete hazard to myself and everyone around me get the test folder sorting nicely.
I also needed exiftool to get the file creation dates I wanted.
To make the Automator thing:
- make the sorting folder where you like (mine is just in my home directory on the same level as the Nextcloud sync folder for no other reason than convenience)
- make a new Folder Action in Automator
- in Folder Action receives files and folders added to dropdown, locate your sorting folder
- put in a Get Specified Finder Items node. If you haven't used Automator before you'll probably get a warning dialogue at some point saying that you need it to test the folder action in Automator and should remove or disable it when using the actual folder to trigger the folder action. However I found that for whatever reason removing or disabling the node stopped the folder action from working at all
- select or drag and drop your sort folder into the box
- put in a Get folder contents node
- put in a Run shell script node
- change Pass input to as arguments
Copy this into the script box. If you run it in Automator it should spit out a bunch of comments to tell you what's going on while you're testing, or it will just do its thing if you just use the folder action.
ALWAYS TEST ON A TEST FOLDER WITH TEST FILES FIRST.
for f in "$@"
do
# use exiftool to get the creation dates we want
# sometimes all exif data is missing in which case we have to fall back
# to a file system create date
#
# the multiple -s gets rid of some labels that exiftool helpfully
# supplies for if you're using it on command line
device=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" "$f")
device2=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal "$f")
exif=$(/usr/local/bin/exiftool -s -s -s -createdate "$f")
fs=$(stat -f "%SB" $f)
# now we're going to break down the dates into year/month/day variables
# and populate starting from the least desirable file system create date
# and hopefully override with the more desirable dates further down
if [[ -z $fs ]] # -z means "empty" and if this first one is empty there's problems XD
then
echo wtf how does $f:t have no fs create date
else
y=$(stat -f "%SB" -t "%Y" "$f")
m=$(stat -f "%SB" -t "%m" "$f")
d=$(stat -f "%SB" -t "%d" "$f")
echo $f:t has filesystem create date $y $m $d
fi
# some of the files I was working with had zeroed exif dates for some reason
# the next few ifs check for empty variables or if they start with four zeroes
# in which case it counts as empty
if [[ -z $exif || ${exif:0:4} == "0000" ]]
then
echo $f:t has no exif date
else
y=$(/usr/local/bin/exiftool -s -s -s -createdate -d "%Y" "$f")
m=$(/usr/local/bin/exiftool -s -s -s -createdate -d "%m" "$f")
d=$(/usr/local/bin/exiftool -s -s -s -createdate -d "%d" "$f")
echo $f:t has exif date $y $m $d
fi
if [[ -z $device2 || ${device2:0:4} = "0000" ]]
then
echo "$f:t has no device2 date"
else
y=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal -d "%Y" "$f")
m=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal -d "%m" "$f")
d=$(/usr/local/bin/exiftool -s -s -s -datetimeoriginal -d "%d" "$f")
echo $f:t has device2 date $y $m $d
fi
if [[ -z $device || ${device:0:4} == "0000" ]]
then
echo "$f:t has no device date"
else
y=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" -d "%Y" "$f")
m=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" -d "%m" "$f")
d=$(/usr/local/bin/exiftool -s -s -s "-Keys:CreationDate" -d "%d" "$f")
echo $f:t has device date $y $m $d
fi
# change this to the path you want the files to be sorted into
# (which is obviously going to be different from your sort folder just in case
# that wasn't clear from the outset)
target=/path/you/want/things/sorted/to/$y/$m/$d
# if the target directory doesn't exist
if [[ ! -d $target ]]
then
echo $target created, moving $f:t
mkdir -p $target # make all the folders
mv $f $target # and move the file to the target folder
# if the target directory does exist
else
echo $target exists
# check to see if there's a file with the same name in there
if [[ ! -e $target/$f:t ]]
then
# if there isn't, move the file
mv $f $target
echo moved $f:t to $target
else
# if there is, we're going to add a number to the file name
echo file called $f:t already exists in $target
n=1
# first we check to see if there is already a filename-nn.ext
# if not, we rename the file and then move it
if [[ ! -e $target/$f:t:r-$n.$f:t:e ]]
then
echo n is $n
mv $f $target/$f:t:r-$n.$f:t:e
echo renamed to $f:t:r-$n.$f:t:e and moved to $target
# if there is a filename-nn already in the folder, we add 1 and try again
# until we get a unique filename and then rename and move the file
else
while [[ -e $target/$f:t:r-$n.$f:t:e ]]
do
echo file called $f:t:r-$n.$f:t:e already exists in $target
(( n++ ))
mv $f $target/$f:t:r-$n.$f:t:e
echo renamed to $f:t:r-$n.$f:t:e and moved to $target
done
fi
fi
fi
done
There's more information on getting bits out of the path (the $f:t etc bits) and the test operators (the -z, -e etc bits) that I found in my travels that helped immensely.
The above can be modified to a photos cleanup script by ticking the Repeat for each subfolder checkbox in the Get folder contents node and adding in a Filter Finder Items node straight after it and before the script node. The filter you then need is Kind is not folder.
I did try this out but it turned out a lot of my older photos have incorrect date information so it didn't work out that well in my case and I'll have to find and fix problems manually on the Nextcloud server when I can be bothered.
There are probably much better ways to write this (including theoretically adding ! to the conditionals and removing all the echos if you just wanted it to do its thing and don't care about feedback) but I suck at coding, it took me way too long to come up with this and even longer to try and fail to get the cleanup version working.
I have tested it out with a manual export (because after importing everything into Photos the export smart album of course contained a lot of things because a lot of things had been imported "in the last day") and everything seems to be working as it should. The only minor panic I had was that it exported heic files (probably because I ticked the box to export unmodified originals), but digikam is able to read them.
Can't say the same about the Nextcloud webapp as browsers haven't caught up yet but that's fine XD
Now I can get back to art and everything else I've been neglecting while getting this sorted x_x
This post also appears on my blog.
sleep is one of the many things I've been neglecting x_x
Want a crowd character? If you interact frequently with me here (either on my posts or your posts or better yet both!) or on one of the linked Discord channels, all you have to do is let me know and tell me what you'd like :) Everything you need to know is (probably) here and if it isn't just ask (sorry the link goes offsite to my blog where the layout is how I want rather than the one here which is only here because the original one was).
Thanks for looking! ^_^
Come hang out on Discord with us!
#teamaustralia on PAL | ANZ | Be Awesome | artbees
I for one missed your comments.
I am also trying to sort my photos after a hard disk crash caused by little fingers turning off my computer while it was just starting up. Luckily I have them all backed up online but it does mean I have to download them all again to have a local copy.
I still need to work on my naming and folder conventions, For dealing with images I usually rely on ACDSEE but I recently found an open-source one called Faststone image viewer, it has batch conversion tools also. Maybe check it out.
I love the fact that you can do coding as well.
^_^
Oh no little fingers they're lucky they're so cute DX phew yay for backups, this is why I have so many redundancies XD (I have two copies of my project on the big rig, one in the working folder and one on the nextcloud sync folder which are kept in sync with FreeFileSync, two copies of the photos on the small rig, one in the internal Photos thing and the one in the nextcloud sync folder with exports from Photos going straight into the nextcloud sync folder, and also have backups from the big rig which does both a project backup and an entire homedir backup including the nextcloud sync folder to an external drive, which gets swapped over weekly or so with another one that's at my aunty's place XD)
As shown in the script I just use /y/m/d so for example my latest photo imported from my phone on the 20th of November is in 2020/11/20/IMG_0301.HEIC. Makes it slightly easier for me to find things using file browsers or digikam on the big rig as I usually have some idea of at least what year the photo was taken in XD
One of the things I do like about Photos is even if it's pretty useless for most other things, it's great at automatically sorting and doing this pretty mosaic display thing (pretty is just as important as functional for me) so it's one less thing for me to think about.
Had a quick look at Faststone, looks like Windows only? I have Manjaro and MacOS and basically avoid Windows like the plague because that's the one I can break by being in the same area too long XD
Never sure if my cheap hacks can count as coding ;D
oh ps you get to be today's experiment to see if I can do this yet:
!ENGAGE 10
ENGAGEtokens.Awesome!
I must admit that I scanned this post, as it felt like dev talk ( 0s and 1s ) and yeah, I have been missing your comments ( not just yours ), as you are one of the most faithful commenters on my feed and the feed of many.
Glad to find out that this was the reason for your 'absence'. It reminded me somewhat of editing on a MAC ( in 2008 and earlier in that decade ) and realizing how much different these machines are than the Windows PCs that I grew up with. It's like your entire brain needs to be rewired.
Big hug,
Vincent
On a pure technicality I'm wondering if you meant skimmed as that means you read over it really quickly whereas if you're scanning you're still reading quickly but looking for specific things XD
crawls back into hole
Most of it was techy nonsense but it was stuff that needed to be done for my photo collection shenanigans to work :)
Unfortunately some other crap has come up that we could have really done without so I won't have much prog to blog again >_<
but I can at least comment when my brain cooperates
!ENGAGE 10
ENGAGEtokens.Congratulations @ryivhnn! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCongratulations @ryivhnn! You received a personal badge!
Wait until the end of Power Up Day to find out the size of your Power-Bee.
May the Hive Power be with you!
You can view your badges on your board and compare yourself to others in the Ranking
Congratulations @ryivhnn! You received a personal badge!
Participate in the next Power Up Day and try to power-up more HIVE to get a bigger Power-Bee.
May the Hive Power be with you!
You can view your badges on your board and compare yourself to others in the Ranking