Tardigrades Will Inherit The Earth (output in ffmpeg command)

True - fair dinkum mate :smiley: - tardigrades… Considering I was ALWAYS late for school (when I wasn’t wagging / playing hookie / truant) I can relate to tardigrades :smiley:

OK - I now have my script to convert a folder of FLAC files to a new folder of mp3 files… Slowly housekeeping all the duplicates (i.e. delete an mp3 album if I already have it in FLAC).

And - just decided to do some housekeeping on my digital movie “offsite backup for archival purpose only” :smiley: collection - I have dozens of them over 4 GB in filesize, some were close to 8 GB…

So - I’ve been using “ffmpeg” (I can’t imagine life without it - especially as my favourite movie player [mpv] use ffmpeg) to resample overly large video files using the libx265 codec (which is an upgrade of H.264 I believe) and changing the compression ratio (lossiness) - and watching the re-sampled video on my QHD 32" 165 Hz monitor about 2 feet away - having BOTH of them side-by-side - I can’t tell the difference - in some cases I’ve made a 10 fold saving! An 8 GB mkv is now between 8-900 MB and it looks just as good… For special stuff - I’m setting a slighly less lossy (-crf 24) value… I think the default is 23 - I might actually try 23 next time around and see if I still get any space savings…

Anyway - I got sick of searching back through my shell history - and I ended up writing a shell script to do it for me :

╭─x@titan ~  
╰─➤  bat ~/bin/reducto.bash                                                                                                                                  
───────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: /home/x/bin/reducto.bash
───────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ #!/usr/bin/env bash
   2   │ # Reduce a video file by #FACTOR
   3   │ PROG=$(basename $0)
   4   │ BNARY=ffmpeg
   5   │ echo "Usage : "
   6   │ echo -e "\t $PROG \$videofile \$factor"
   7   │ echo -e "\t\t where \$vidofile exists and : "
   8   │ echo -e "\t\t  $factor is some value between 20 (lowest) and 51 (highest) compression..."
   9   │ # defaults to using x265 format...
  10   │ if [ $# -lt 2 ] ; then
  11   │     echo "this $PROG expects two arguments..."
  12   │     exit 1
  13   │ fi
  14   │ VID="$1"
  15   │ FCT=$2
  16   │ if [ ! -f $VID ] ; then
  17   │     echo "$VID doesn't seem to exist...."
  18   │     exit 1
  19   │ fi
  20   │ if [[ "$FCT" =~ ^[0-9]+$ || "$FCT" =~ ^[-][0-9]+$  ]] ; then
  21   │     echo "looks like an integer then...."
  22   │ else
  23   │     echo "$FCT is NOT an integer..."
  24   │     exit 1
  25   │ fi
  26   │ if [ $FCT -gt 51 ] ; then
  27   │     echo "$FCT out of range - should be less than 52!"
  28   │     exit 1
  29   │ fi
  30   │ # EXT=$(echo $VID | awk '{print $NF}')
  31   │ # echo "${filename%.*}"
  32   │ NEWVID="${VID%.*}"
  33   │ echo Old : $VID
  34   │ echo New : $NEWVID
  35   │ echo "$BNARY -i $VID -vcodec libx265 -crf $FCT $NEWVID-x265-crf$FCT.mp4"
  36   │ $BNARY -i "$VID" -vcodec libx265 -crf $FCT "$NEWVID-x265-crf$FCT".mp4
───────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

AND I managed to write a shell script without any SWEARING in it too! I might change it a bit later to accept a 3rd argument, filetype by extension - i.e. mkv, mpg, avi as well as mp4…

That NEWVID="${VID%.*}" stuff is to lose the existing filename extension…

Anyway - about tardigrades - some of the output (several times - for each “stream”) from ffmpeg shows (that is quite possibly actually in the library - not the ffmpeg binary??? "mkvmerge or libx265) :

 Stream #0:0: Video: hevc (hev1 / 0x31766568), yuv420p(tv, bt709, progressive), 1280x688 [SAR 172:173 DAR 320:173], q=2-31, 23.98 fps, 24k tbn (default)
    Metadata:
      BPS-eng         : 3977147
      DURATION-eng    : 02:00:48.992000000
      NUMBER_OF_FRAMES-eng: 173802
      NUMBER_OF_BYTES-eng: 3603788774
      _STATISTICS_WRITING_APP-eng: mkvmerge v21.0.0 ('Tardigrades Will Inherit The Earth') 64-bit
      _STATISTICS_WRITING_DATE_UTC-eng: 2019-02-12 08:35:12
      _STATISTICS_TAGS-eng: BPS DURATION NUMBER_OF_FRAMES NUMBER_OF_BYTES
      encoder         : Lavc58.134.100 libx265
    Side data:
      cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A

I love seeing humorous little “asides” in things like this!


Anyway - if I keep up this sort of house keeping the existing disks in my NAS should last at least another few years - I was under 1 TB free late last year - I now have 1.3 TB free!

2 Likes

I can only run one of these jobs at a time - tried to run two earlier today and slowed my desktop to a crawl :
bashtop output :


Post Script 1
I’m going to have to update my script to specify how many threads to use - just tried to do it now - and just one instance made my desktop crawl… Doesn’t really matter how long it takes - so long as it gets done… I can kick one off just before I go to bed for the night…


Post Script 2
Hmmm - just tried “ffmpeg -threads 4 ...” then “ffmpeg -threads 2 ...” - still crippled my desktop machine… didn’t seem to make a difference…
So - I reckon I might break out my old Phenom X6 system - install headless Ubuntu 24.04 server on that - and use that to re-sample video files…

In the first instance - Post Script 1 - I had to actually use TermUX on my phone to SSH and kill the ffmpeg job as I couldn’t get terminator or anything to respond…


Post Script 3
Hmmm - why use an old AMD Phenom II X6 - I have two MacBook M1? … Running it now with 2 threads and I can barely notice it impacting the Mac desktop environment… I might turn of the “-threads 2” setting completely and just let it kill the performance on the Mac… Probably get the script to have a 3rd argument - i.e. whether to use “threads” argument or not… and how many threads… The M1 has 8 cores…

By any chance… do you have hardware to assist h265 enciding? That would speed up the process a lot…

1 Like

Hmmm - I wonder if I can use my GPU?

Radeon RX 6600/6600 XT/6600M (with 8 GB of DDR5 - not sure what AMD equivalent of CUDA is - so no idea how many GPU cores I have…

Plenty of other options to explore - but - I’m running it on all 8 threads on my work MacBook Pro M1 - and doesn’t seem to be flatlining like it does on Pop!_OS 22 with AMD Ryzen 7 CPU…

I hope I don’t have to do any compiling to get ffmpeg and libx265 to use my GPU as well as CPU… But looks like that is an option I can explore… I reckon would probably easier and more straightforward on NVidia with proprietary drivers to use CUDA cores for offloading onto…

Threads will not help if it is I/O bound.
Can you throttle the I/O?

Depends also on your ffmpeg build.

I get:

ffmpeg -encoders | grep hevc
ffmpeg version 5.1.5-0+deb12u1 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 12 (Debian 12.2.0-14)
  configuration: --prefix=/usr --extra-version=0+deb12u1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libglslang --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librist --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --disable-sndio --enable-libjxl --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-libplacebo --enable-librav1e --enable-shared
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
 V....D libx265              libx265 H.265 / HEVC (codec hevc)
 V....D hevc_nvenc           NVIDIA NVENC hevc encoder (codec hevc)
 V..... hevc_qsv             HEVC (Intel Quick Sync Video acceleration) (codec hevc)
 V..... hevc_v4l2m2m         V4L2 mem2mem HEVC encoder wrapper (codec hevc)
 V....D hevc_vaapi           H.265/HEVC (VAAPI) (codec hevc

So I could use nvenc with nvidia (if I hadn’t discarded it :smiley: ), the quicksync with intel gpu, and VAAPI.
I use the intel quicksync nowadays for transcoding stuff, it’s lightning fast and good quality - requires non free va driver installed.
If I had only the Radeon, I’d try to go with the vaapi codec,

2 Likes

Cool - so it looks like I have a choice of hevc_vaapi on my AMD Linux systems - and probably “hevc_videotoolbox” on my M1 Macs (if I can ever get homebrew working again on my work MacBook - ffmpeg converted a video file for me last night - went to use it again this morning and nothing from homebrew is working [e.g. not even vi / vim]). I don’t know if using GPU on Ryzen with Vega (i.e. Radeon APU) is stealing threads from the CPU (i.e. robbing Peter to pay Paul) or not…

I’m going to make a copy of some smaller files somewhere and run a few benchmarks / test runs to get a feel for it…

I now have this in my script (if it gets a 3rd argument it assumes that’s the number of threads - if not the script defaults to 8) :

echo "$BNARY -threads $THRD -i $VID -vcodec libx265 -crf $FCT $NEWVID-x265-crf$FCT.mp4"
$BNARY -threads $THRD -i "$VID" -vcodec libx265 -crf $FCT "$NEWVID-x265-crf$FCT".mp4

If I try to have “-threads N” as a $VAR in that - it treats it as a double space and doesn’t proceed for some reason I can’t be arsed to figure out…

2 Likes

It would seem they can ‘freeze-dry’ themselves into a state or rest. Bacteria can do that, but I did not know any multicelled animal could do it.
The chemistry of life is fascinating… I did not think so when I almost failed my biochem exam.

2 Likes

Hmmm - maybe not such a good idea?

Trying it on my Pi5 right now - 4 arm64 cores and 8 GB RAM… No idea how long it will take…

Hmmm - a while I reckon :smiley: - a LONG while …

My Ryzen Pop! machine does about 68 frames per second…

both my M1 macbooks about avg between 20-40 fps…

The Pi 5? about 4.4 fps :smiley:

I don’t reckon the AMD Phenom II X6 would be that sluggish… So the arm64 cores in a Pi are still a far cry from the arm64 cores in Apple Silicon :smiley:

1 Like

On Rpi I would probably try hevc_v4l2m2m.
So far never tried…

As for benchmark, intel quicksync encoding can do near 400 fps at full hd resolution on my side. It’s for h264 though, did not need h265 yet…

1 Like

From what I’ve read - h265 gives better quality for less size…

How do you use both a GPU and a codec like libx264 at the same time on the same CLI?

I can’t figure that out… I reckon I’d get better throughput on the Macs if I used “videotoolbox” - but not sure how to use both that, and libx265 on the same command…

Yes, but there’s no free lunch. h265 requires much more computing power to encode :wink:

I have no idea…

1 Like

When I try to use hevc_vaapi (instead of libx265) I get errors :

ffmpeg -threads 2 -i Alatriste-BRRIP-1080p-x264.mpv -vcodec hevc_vaapi -crf 24 Alatriste-BRRIP-1080p-x264-vaapi-crf24.mp4

...
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_vaapi))
  Stream #0:1 -> #0:1 (aac (native) -> aac (native))
Press [q] to stop, [?] for help
Impossible to convert between the formats supported by the filter 'Parsed_null_0' and the filter 'auto_scaler_0'
Error reinitializing filters!
Failed to inject frame into filter network: Function not implemented
Error while processing the decoded data for stream #0:0
Conversion failed!
╭─x@titan ~/Videos/Movies/2006-Alatriste  
╰─➤  ffmpeg -encoders |grep hevc                                                                                                                  1 ↵
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
  built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
... SNIP ...
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
 V..... libx265              libx265 H.265 / HEVC (codec hevc)
 V..... nvenc_hevc           NVIDIA NVENC hevc encoder (codec hevc)
 V....D hevc_nvenc           NVIDIA NVENC hevc encoder (codec hevc)
 V..... hevc_qsv             HEVC (Intel Quick Sync Video acceleration) (codec hevc)
 V..... hevc_v4l2m2m         V4L2 mem2mem HEVC encoder wrapper (codec hevc)
 V....D hevc_vaapi           H.265/HEVC (VAAPI) (codec hevc)

But this works :
ffmpeg -threads 2 -i Alatriste-BRRIP-1080p-x264.mpv -vcodec libx265 -crf 24 Alatriste-BRRIP-1080p-x264-x265-crf24.mp4

Reckon I’ll just keep plodding along using libx265… and paying for my lunch (which is usually free - cause I usually fast all day until dinner time - I’ve lost 10+ kgs in 18 months - as well giving up sugar in my coffee).


Post Script 1:
This worked :
ffmpeg -vaapi_device /dev/dri/renderD128 -threads 2 -i Alatriste-BRRIP-1080p-x264.mpv -c:v hevc_vaapi -vcodec libx265 -crf 24 Alatriste-BRRIP-1080p-x264-vaapi-crf24.mp4
(or it didn’t crash out anyway - or not yet)
Started of getting ~140 fps - but now dropped below 30…
Might do better if I didn’t limit threads to 2 maybe?

1 Like