ChkMagick

HOWTO check for new files, process and move..

Background

I was to find a solution where files copied to a folder automatic was transformed. In this case it was images, pictures. Copied to a special folder, transformed and rotated instant. This auto-chk-tool with scripting features was hard to find...

One solution

This simple tool ChkMagick installs as a service in windows, listens to one or more folders, and performs one or more actions to the files which are placed in those folders.

It uses the windows FindFirstChangeNotification which means it does use very little resources. (A nonsense benchmark, Idle time 4886h, ChkMagick 6s)

Every [Section] in the ini file gets its own thread. Within a section files are handled one at a time, the files waiting in the From= folder is the queue.

Download and install

Administration

Log files

If there is a log\ folder next to chkmagick.exe the logs are written there, otherwise next to the exe. A new set of files is started every month, yymm.
FileContent
ChkMagick-yymm-log.txtmoved files, executed commands, times, depending on LogL=
ChkMagick-yymm-error.txterrors, timeouts, duplicates, missing folders
ChkMagick-yymm-service.txtstart / stop and the configuration read for every section
ChkMagick-yymm-profile.txtfile date (LogL>2) and the full exif/iptc profile (LogL>3) for each file, use it to find the names of metadata %parameters
ChkMagick-section_n.tstoutput from Tstn=
ChkMagick-section.inilist of processed files (duplicate test), saved on stop, read on start

How a file is processed

  1. A change notification for the From= folder (and subfolders if SubT=1) wakes the section. Every 15s it also checks again after an earlier i/o error.
  2. The file must be ready: same size (>0) between two checks, waiting 1+2+3.. seconds, max ~13 minutes. Zero length files older than 30 minutes are deleted.
  3. MaxProc=: if too many processes are running the section waits here, the file stays in the queue.
  4. Sleep= ms is waited (not the first round after start).
  5. Filename prefixes !p123! etc are read and removed.
  6. SecretKey=: the time hash is tested, invalid files are moved to SecretDest= or deleted.
  7. Destination folder is selected by Extn=/ToDn= or To=, filename by File=.
  8. Duplicate test (name, project id, size and Crc32). Duplicates go to DuplDest= and run MvApp1..3, unless Dupl=1.
  9. The file is moved, an existing file is never overwritten (name_1.ext, name_2.ext ...).
  10. For every n=1..16: Copn, Appn (if Iffn and size limits ok), Pushn, Tstn.
  11. Drop=1 deletes the moved file.
  12. When all files in the folder are done, MdToApp= runs if a new destination folder was created.

ChkMagick.ini file format

chkmagick.ini is read when the service starts.
  1. if there is a chkmagick-computername.ini it is prefered
  2. if above does not exists it checks for chkmagick.ini
  3. if chkmagick.ini does not exists, the service stops

[Setup]

From the [Setup] section you set default values for all other sections.
[Setup]DescriptionDefaultExample
Root=where the commands reside when Appn= has no pathRoot=c:\programs\imagemagick
Prio=process priority class Low|Normal|High|Realtime (also Idle|-1|0|1, låg|hög)Low=Idle
Time=max time to wait for each command, ms (min 1000)18000
Kill=0|1, terminate the process if not finished within Time0
Sleep=wait before each file is processed, ms1000
LogL=
LogLevel=
log level 0..9, 0=one line per command, 1=normal, 3=config+profile, 5..9=debug1LogL=2
File=
FileNameMask=
default filename mask in the To folderFile=%srcfn_simple
FileID=
FilenameID=
default map for %filenameid
MinFileName=
MinFileNameLength=
shorter filenames are extended with _yymmdd_random12MinFileNameLength=0
MaxFileName=
MaxFileNameLength=
longer filenames are cut60
FileNameYearCheck=how many years back a date in the filename is trusted, see filedates10
MinFileSize=min filesize to run commands, file is still moved. 16384, 100KB, 10MB-1 = no check
MaxFileSize=max filesize to run commands-1 = no check
SubT=
SubTree=
0|1, also listen to subfolders of From0
App=
Cmd=
Dir=
default Appn/Cmdn/Dirn when not given in a command
SecK=
SecretKey=
secret key for the time hash testSecretKey
SecT=
SecretTime=
seconds per time hash window, 0 = no test0
MaxProc=newsee MaxProc0 = no limit
MaxProcName=
MaxProcWait=1000
MaxProcTimeout=0

Note, Dupl=, DuplDest= and LogRotate= are not read from [Setup], set them in each section.

[Section]

For every folder to listen to you define a new section. All [Setup] keys above can be overridden here.
[Section]DescriptionDefaultExample
Folders
From=the folder to listen to, when a file is copied/created here the process begins.
MUST be an existing folder.
From=c:\upload\img
SubT=0|1, also listen to subfolders[Setup]SubT
To=the folder to move the file to, created if it does not exist. %parameters can be used, ~ is replaced by \To=c:\arkiv\%yyyy\%mm
Extn=list of extensions, n=1..16, first match decides the folderExt1=.7z;.zip;.rar
ToDn=
ToDirn=
move to this folder if extension in Extn, otherwise To=ToD1=c:\upload\zip
MdToApp=command to run when a new destination folder was createdMdToApp=c:\cmd\ftpdirs.exe
MdToCmd=parameters to MdToApp, the new folder is added last
MdToSto=standard output (append) for MdToApp
Filenames
File=
FileNameMask=
filename mask in the To folder, the extension is kept[Setup]FileFile=%srcfn_simple%_U%now12
FileID=
FilenameID=
map for %filenameid, name=id;name2=id2. The first name found in the filename gives the id. If no match, %filenameid=0 and the rest of the commands are skippedFileID=kvitto=1;faktura=2
MinFileNameLength=
MaxFileNameLength=
see [Setup][Setup]
FileNameYearCheck=see [Setup][Setup]
Duplicates
Dupl=0|1. With 0 a file already processed (name, project id, size and Crc32, list cleared every night) is moved to DuplDest and the commands are not run.
With 1 the commands are run anyway. If an identical file already exists in To the new file is deleted (when DuplDest is empty) and the commands are run on the existing file.
0
DuplDest=folder for duplicateschkmagick\duplicates\DuplDest=c:\dupl
Comp=compare with this folder instead of To (with Dupl=1). If a file with the same name but not the same content exists the filename gets _n.
Use it when files move in stages and the name must be unique in the final folder.
MvAppn=n=1..3, commands to run when a file is moved to DuplDest. Same keys as below with prefix Mv, MvCmdn, MvDirn, MvStin ...
Processing
Time=max time for each command, ms[Setup]TimeTime=200000
Kill=0|1, terminate the process when Time is reached. With 0 the process keeps running in the background and the next file starts[Setup]Kill
Sleep=wait before each file, ms[Setup]Sleep
Prio=default priority for the commands[Setup]Prio
Root=path for commands without path[Setup]Root
MinFileSize=
MaxFileSize=
size limits to run commands[Setup]
Exif=0|1, set the filedate from the exif date of the moved file0
Drop=0|1, delete the file after all commands0
MaxProc - limit running processes new
MaxProc=max number of running processes with the same exe name (all processes on the computer, not only from this section). If reached, wait until one has finished. The files wait in the From folder.0 = no limitMaxProc=6
MaxProcName=exe name(s) to count, separated by ; or ,. Used for all commands in the section.exe of AppnMaxProcName=mariadb.exe
MaxProcWait=how often to count again while waiting, ms (min 100)1000MaxProcWait=2500
MaxProcTimeout=max wait per file, ms. Then the command starts anyway (note in the error log)0 = wait foreverMaxProcTimeout=0
Security, time hash
SecretKey=if set together with SecretTime, the first 1KB of the file must contain a valid time hash (current or previous time window)[Setup]SecretKey
SecretTime=seconds per time window[Setup]SecretTimeSecretTime=1800
SecretDest=
InvalidDest=
folder for files without a valid hash, if empty the file is deleted
Logging
LogL=
LogLevel=
log level for this section[Setup]LogL
LogR=
LogRotate=
rotate the log file when larger than, min 8KB. 16384, 100KB, 8MBno rotationLogRotate=8MB
Commands, n=1..16, run in order for every file
Appn=the command to run, without path Root= is used[Section]AppApp1=convert.exe
Cmdn=parameters to the command, with %parameters[Section]CmdCmd1=%f -thumbnail 120 %fil.jpg
Dirn=the folder to run the command in, created if it does not existpath of Appn
Prin=
Prion=
priority for this command[Section]Prio
Iffn=run only if the original filename matches this regular expressionIff1=(?i)^Kvitto_|^Receipt_
Stin=
StandardInputn=
file for standard input. If it does not exist (after ~1s) the command is not runSti1=%f
Ston=
StandardOutputn=
file for standard output, appendedSto1=c:\tmp\%fnseu4.text
Copn=
Copyn=
copy a file before Appn is run, source destination separated by space, ; or ,. An existing destination is not overwrittenCop1=c:\dir\file.txt c:\dir\%yyyy\%fil.txt
Exin=
Exifn=
read metadata (%px, %py, exif ...) from this file instead of the moved fileExif8=c:\arkiv\s\%fil_ext1
Pusn=
Pushn=
add text to the push string, read later with %popPush7='%fno','%fil'
Puan=
PushAn=
same for %popaPushA8=,%px,%py
Pubn=
PushBn=
same for %popb
Tstn=
Testn=
write the replaced text to ChkMagick-section_n.tst, to test %parametersTst1=%fil %yyyy %px

A key Appn with only Cmdn uses App= from the section, that way one program can be called many times:
App=convert.exe + Cmd1=... Cmd2=....
tcc.exe and cmd.exe are started in a new console. Some programs lose the first parameter, then use Cmdn=0 param1 param2.

[DateTime]

Optional, sets the date/time format used by %fdat etc. if the server locale is wrong.
CurrencyString CurrencyFormat NegCurrFormat ThousandSeparator DecimalSeparator CurrencyDecimals DateSeparator ShortDateFormat LongDateFormat TimeSeparator TimeAMString TimePMString ShortTimeFormat LongTimeFormat ShortMonthNames1..12 LongMonthNames1..12 ShortDayNames1..7 LongDayNames1..7

Filename prefixes

The uploader can send ids in the filename, they are removed before the file is moved: !p123!u45!mtext!photo.jpg ⇒ photo.jpg
PrefixSetsRead with
!p123!project id%projid
!s123!sub project id%delpid
!o123!organisation / part id%partid
!u123!user id%userid
!mtext!metadata, can be repeated%metadata, %meta
!dtext!directory tag (xdtext), can be repeated%directories, %meta
The ids must be numbers. If the project id is missing it is taken from a source folder like proj_123 or proj_123.4, part id from part_123 and user id from user_123.

Replacable parameters

In To, File, Comp, Cmdn, Dirn, Stin, Ston, Copn, Exifn, Pushn, Tstn, SecretDest you can use the following parameters. Names are lowercase letters, digits and _.
  • end a name with % to put text directly after it, %date6%_%time4 ⇒ 141224_1445
  • %% gives one %
  • an unknown name is replaced with nothing
A picture file taken 2005-apr-08: img_0123.jpg
is dropped 2014-dec-24 14:45:15 in
c:\home\web\upload\img\project_23\
The section below handles the file for the "result" below
[Nav-img]
From=c:\home\web\upload\img
SubT=1
To=c:\home\nav\web\upload\img
File=%yymmdd_%fnr_%r%fc
%parDescriptionResult
Source (original) file
%src
%fo
original file with pathc:\home\web\upload\img\project_23\img_0123.jpg
%srcfne
%fno
original filename with extensionimg_0123.jpg
%srcfn
%fn
original filename without extensionimg_0123
%srcextoriginal extension, lowercase.jpg
%srcpathoriginal pathc:\home\web\upload\img\project_23\
%src_unix
%srcfn_unix
%srcpath_unix
same with /c:/home/web/upload/img/project_23/
%srcfn_simple
%fns, %fnh
simple filename, only a..z 0..9 _ - ! ( ), åäö ⇒ aao. With SubT=1 the last subfolder is added as prefix folder_img_0123
%srcfne_simple
%fnse, %fnhe
same with extensionimg_0123.jpg
%src_simplesimple filename, no folder prefiximg_0123
%fnseu2..%fnseu8simple filename + _ + unique string + extensionimg_0123_x8y4.jpg
%yymmddfnsfile date + simple filename, old dates in the name are removed050408_img_0123
%src1filename before the first ! $ @
%src2second part of the filename between !
%srcfn_nr
%fnr
filenumber of the filename, max 16img0123
%srcfn_nr_uc
%fnru
same, uppercaseIMG0123
%fnr8, %fnr12, %fnr24, %fnr32, %fnr48filenumber max 8..48 chars, %fnrunn uppercase
also %srcfn_nrnn, %srcfn_nrnn_uc
%srchash2..6
%fh2..%fh6
small hash from filename + filesize, 2..6 charsa3
%srcdate
%fdat
file date and time2005-04-08 12:30:00
Source folder parts (c:\home\web\upload\img\project_23\)
%srcpath2
%fd
the folder of the fileproject_23
%srcpath3
%fdb
parent folderimg
%srcpath4parent of parentupload
%srcpath_id
%fdd
last part of the folder after _23
%srcprojidfirst folder in the path with only digits
Destination (moved) file
%dst
%f
moved file with pathc:\home\nav\web\upload\img\050408_img0123_600.jpg
%dst0
%f0
same, for .gif and video [0] is added = first frame for ImageMagick...\anim.gif[0]
%dstfne
%fme
filename with extension050408_img0123_600.jpg
%dstfn
%fm
filename without extension050408_img0123_600
%dstext
%fe
extension, lowercase.jpg
%dstpath
%fp
pathc:\home\nav\web\upload\img\
%dst_unixfile with /c:/home/nav/web/upload/img/050408_img0123_600.jpg
%dstfn_unix
%fu
file with /, no extensionc:/home/nav/web/upload/img/050408_img0123_600
%dstpath_unix
%fpu
path with /c:/home/nav/web/upload/img/
%filthe target filename, without path or extension050408_img0123_600
%filetarget filename + extension050408_img0123_600.jpg
%ext1smart extension: .png for gif/bmp/ico/png/tif, else .jpg.jpg
%ext2same, but large png/tif (>1920x1080) gives .jpg.jpg
%fil_ext1
%fil_ext2
%fil + %ext1 / %ext2050408_img0123_600.jpg
%dst2
%dst2fn, %dst2fne, %dst2ext, %dst2path
%dst2_unix, %dst2fn_unix, %dst2path_unix
destination without !p123! prefixes
Ids and archive
%projid, %delpid, %partid, %useridids from prefixes or folder23
%metadata, %directories, %metatext from !m..! and !d..!, %meta = both
%arkiv
%arkivdos
archive folder: proj\projid (projid>7), part\partid (partid>7), user\userid, super\1 (id=1) or pub\0proj\23
%arkivunixsame with /proj/23
%filenameid
%fnid
id from FileID= map, 0 skips the rest of the commands1
Image and metadata (from the moved file or Exifn=)
%width
%px
pixel width, null if unknown1024
%height
%py
pixel height, null if unknown768
%rot
%srcimrotate
ImageMagick rotation from exif orientation-rotate 90
%r
%srcexifrotate
exif orientation 0,6(=90),8(=-90)6
%latitude
%lat
GPS latitude, negative for S59.3293
%longitude
%lng
GPS longitude, negative for W18.0686
%magicktitletitle from exif/iptc/jpeg comment
%magickcamcamera, exposure, f-number, ISO, focal lengthCanon EOS 5D | 1/125s | f/8 | ISO100 | 50mm
%magickcopyartist, author, copyright
%magicktagskeywords and categories
%magickindex
%magickindex255
all words above for a search index, 255 = max 255 chars
%resnnnresolution (dpi) to get nnn pixels on the longest side, for .pdf/.eps/.ps-r%res600
%tagnameany exif/iptc/eps value in -profile.txt, written lowercase. Quotes and \ are escaped for sql%gpsimagedirection
%creationdate
File date (see filedates)
%yyyyyear, 4 digits2005
%yyyear, 2 digits05
%mmmonth 01-1204
%ddday 01-3108
%yymmdddate, 6 digits050408
%hhhour 00-2312
%nnminute 00-5930
%sssecond 00-5900
Current date and time
%year2yy14
%year4yyyy2014
%date4yymm1412
%date6yymmdd141224
%date8yyyymmdd20141224
%date10yyyy-mm-dd2014-12-24
%time2hh14
%time4hhmm1445
%time6hhmmss144515
%time8hh:mm:ss14:45:15
%now8yymmddhh14122414
%now10yymmddhhmm1412241445
%now11yymmdd-hhmm141224-1445
%now12yymmddhhmmss141224144515
%now13yymmdd-hhmmss141224-144515
%now16yyyy-mm-dd_hh-mm2014-12-24_14-45
%now19yyyy-mm-dd_hh-mm-ss2014-12-24_14-45-15
Unique, random and counters
%unique2..%unique8unique string, the same for all commands of one filex8y4
%uniqueint
%uniquei
unique number, the same for all commands of one file1804289383
%random2
%random3
%random4
new random string every time, 00..zz / 000..zzz / 0000..zzzzx8y
%counter
%fc
counter per section 00..ZZ00
Push and pop
%popall Pushn= texts for this file so far
%popclearsame, and clear it
%popa
%popb
PushAn= / PushBn= texts
Other
%content
%fi
content of the file on one line, max 32KB
%usr
%user, %username
windows user of the serviceSYSTEM

Filedates

The file date used by %yyyy, %mm ... is taken from, in order:
  1. exif date for images (if it is not more than 100 days after a date in the filename)
  2. CreationDate for .pdf, .eps, .ps
  3. the file's last modified date (local time), last saved date in the document summary info or a date found in the filename
If no exif or pdf date is found and the filename contains a date, like yymmdd_filename.ext or filename_yyyy-mm-dd.ext, that date is used when it is within FileNameYearCheck= years.

Note on "filenumber"

Examples

Images

This example handles the transformation of pictures after uploaded to a webserver, either by http, ftp or file copy. It uses the ImageMagick package to transform pictures and saves the information in a MariaDB database.
[Setup]
Root=C:\Programs\ImageMagick
Time=12000
LogL=2

[Nav-img]
From=c:\home\web\upload\img
To=c:\home\web\pic\o\%yyyy\%mm
File=%yymmddfns%_%fh4

# thumbnail 120x120
App1=convert.exe
Cmd1=%f0 -auto-orient -thumbnail "30000@" -gravity center -crop 120x120+0+0 +repage _
 -background white -flatten -quality 88 %fil.jpg
Dir1=c:\home\web\pic\t\%yyyy\%mm

# small screen
App2=convert.exe
Cmd2=%f0 -auto-orient -resize 600x700 -strip -quality 80 %fil_ext1
Dir2=c:\home\web\pic\s\%yyyy\%mm

Push3='%fno','%fil','%yyyy-%mm-%dd %hh:%nn:%ss','%latitude','%longitude','%magicktitle'
Exif4=c:\home\web\pic\s\%yyyy\%mm\%fil_ext1
PushA4=,%px,%py

App5=C:\Program Files\MariaDB 11.8\bin\mariadb.exe
Dir5=C:\Program Files\MariaDB 11.8\bin
Cmd5=-D web -u chk -ppw -e "CALL ImgsImport(%pop%popa);"

Sql import queue with MaxProc

Max 6 mariadb.exe on the server, a hung import is killed after 200s. New files wait in the queue folder.
[sql-queue]
SubT=1
From=c:\ftp\queue
To=c:\ftp\proj\%srcpath2\done
File=%srcfn_simple%_U%now12
Dupl=1
Time=200000
Kill=1
MaxProc=6
MaxProcName=mariadb.exe
MaxProcWait=2500
MaxProcTimeout=0

App1=C:\Programs\ConvMPKL\ConvMPKL.exe
Cmd1=0 -u=1 -p=%srcpath2 -t="%fno" %f %fil.sql
Dir1=c:\ftp\proj\%srcpath2\done

App2=C:\Program Files\MariaDB 11.8\bin\mariadb.exe
Dir2=c:\ftp\proj\%srcpath2\done
Cmd2=-D ndh -u chk -ppw --default-character-set=latin1 -e "source %fil.sql"

Send mail with standard input

[mail-pickup]
From=c:\chkmagick\pickup
To=c:\chkmagick\done
File=%date6%_%time4%_%random4%_%fn
App1=c:\programs\command\msmtp.exe
Cmd1=-t
Sti1=%f

Versions / Change-Log

2026-okt-26, version 2.63
MaxProc=, MaxProcName=, MaxProcWait=, MaxProcTimeout= - wait while too many processes are running, the queue builds in From
2026-feb-25
SecretKey=, SecretTime=, SecretDest= - time hash test in first 1KB of file
LogRotate=, thread safe logging
2025-dec-14
more robust test that a file is completely written before it is processed
2025-mar-18
FileID= and %filenameid
2024-mar-14
file date in local time zone
2023-dec-14
subfolder prefix in %srcfn_simple for SubT=1
2023-feb-07
default App=, Cmd=, Dir= per section
2022-okt-08
%arkiv with part destination
2019-okt-10
!o123! partid prefix
2016-sep-25
%ext1, %fil_ext1
2015-okt-14
Pushn=, PushAn=, PushBn= and %pop, %popa, %popb, %ext2, %fil_ext2
2015-aug-16
Iffn= - run command only if filename matches regex
2015-jul-07
Stin=, Ston= standard input and output, MdToSto=
2014-okt-17, version 2.33
%now variables
2014-jul-04, version 2.32
[Section]Kill option to terminate process if timeout is reached
2013-oct-29, version 2.30
Lots of small fixes, same config-file format, updating the web page with latest version
2011-mar-17
FileNameYearCheck=, MinFileNameLength=, MaxFileNameLength=
2010-aug-12
!p123! !u123! !s123! filename prefixes
2010-apr-08
more commands per section, Exifn=
2009-aug-03, version 1.0
Small File Hashes now implemented
2007-feb-07, version 0.24
Exif= directive now implemented
more robust engine.
2006-jun-27, version 0.??
Copn= directive now implemented
2006-jun-19, version 0.13
Prio= directive now implemented
SubT= directive now implemented
%fd implemented
2006-jan-16, version 0.10.1
Comp= directive now implemented
2006-jan-04, version 0.9.1
Logfile moved files filename now correct
%fnr and %fnru max 16char
2005-dec-12, version 0.9
Wordwrap lines, end lines with _.
Comments, lines starting with: #, ., rem , [-- , [==
Eventlog support. Start / stop / errors
2005-okt-30, version 0.8
Moveto folder controlable by file extensions
2005-okt-24, version 0.7
Allow readonly files in listen folder, the readonly flag is turned off.
2005-sep-06, version 0.5
%fu, %fpu added
2005-jun-22, version 0.4
Initial release

More info

if you like we can help you with more template options...

More info/questions from: Subject: ChkMagick

Keywords
Server,Windows,NT,2000,XP,2003,2008,Windows7,Win7,Windows8,Win8,Win32,x86,Windows10,Windows2012,Windows2016,Windows2019,Windows2022, Check,Change,Monitor,DirectoryMonitor,FolderMonitor,Queue,MaxProc, HotFolder,Hot,MoveFiles,MoveFile,ChkFolder,ListenFolder,Listen,ChkDir,Dir,Directory,Folder,Copy,Move, FindFirstChangeNotification,ChangeNotification, ChkMagick,ImageMagick,Auto,Automatic,Empty,Notify,Check