S6 init system with s6-66 service manager in Antix-init-diversity spin

[nevj (Neville Jackson)]

@ProwlerGr

I could be wrong, maybe there can be more than one tree enabled

That Obarun page reads like

• you must have global tree
• you can add any number of other trees
• you can run more than one tree at a time
• trees can have dependencies
• it is the dependencies that form a branching structure so that is where the term tree comes from maybe?

Services within *tree* need to be *enabled* , with the [66 enable](https://web.obarun.org/software/66/latest/66-enable.html) command, to be managed by the `start` subcommand.

That statement, from the Obarun tree page, is either wrong
or ambiguous. We started sshd without enabling it.

We need to devise a simple test and demo tree setup.

It looks like developers of 66 have invented this tree concept but are not prescribing how it should be used. That is being left open.

[ProwlerGr]

Well the command

sudo 66 -z tree status -go name,contents

reveals that sshd is automatically added to the “Global” tree.

Also the manual page states

The default tree named global is provided. This tree is automatically created using basic creation configuration … If the global is removed, a simple invocation of a 66 command will create it again.

Obviously the 66 author (like the s6 author) provides mechanism & not policy, leaving it up to the distro & the system admin how they’d want to set-up the trees.

[nevj (Neville Jackson)]

@ProwlerGr

Obviously the 66 author (like the s6 author) provides mechanism & not policy, leaving it up to the distro & the system admin how they’d want to set-up the trees.

Yes, it is like providing a compiler… you dont attempt to outline all the possible uses, you leave that to the user.

Linux is a tool, not a recipe for everything you would ever want to do.

[ProwlerGr]

Another tip for 66, how to configure the boot@system module.

sudo 66 configure -e nano boot@system

Creates & opens a “special” frontend file for the boot@system module.
Once you make edits & save that file running

sudo 66 reconfigure boot@system

Commits those changes.

[nevj (Neville Jackson)]

I am having trouble
After booting the Antix VM today, sshd no longer starts
(I dont have it enabled yet)

root@antix-init-diversity:/etc/66/service# 66 start sshd
signal: info: Successfully started service: sshd-log
s6-svc: warning: /run/66/scandir/0/sshd/notification-fd not present - ignoring request for readiness notification
signal: info: Successfully started service: sshd

root@antix-init-diversity:/home/nevj# 66 status sshd
Name                  : sshd
Version               : 0.0.1
In tree               : boot
Status                : disabled, down (exitcode 255) 0 seconds, normally up, want up, ready 0 seconds
Type                  : classic

I am trying modifying your frontend

$ cat /etc/66/service/sshd
[main]
@type=classic
@description="sshd daemon"
@version=0.0.2
@user=(root)

[start]
@execute=(
    foreground { mkdir -p /run/sshd 
    foreground { chmod 0755 /run/sshd }
    foreground { exec ssh-keygen -A }
    /usr/sbin/sshd -e -D 
)

because I noticed /run/sshd was missing.
That does not seem to help

I cant even run it by hand

root@antix-init-diversity:/home/nevj# /usr/sbin/sshd -D -e
Missing privilege separation directory: /run/sshd

So I made the directory /run/sshd and did chmod 755 /run/sshd
Then it works

root@antix-init-diversity:~# 66 parse sshd
root@antix-init-diversity:~# 66 start sshd
signal: info: Successfully started service: sshd
signal: info: Successfully started service: sshd-log
root@antix-init-diversity:~# ps ax | grep ssh
 1347 ?        S      0:00 s6-supervise sshd-log
 1348 ?        S      0:01 s6-supervise sshd
 1355 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/sshd
11781 ?        Ss     0:00 sshd: /usr/sbin/sshd -e -D [listener] 0 of 10-100 startups
12017 pts/0    S+     0:00 grep ssh

So

• why did the directory /run/sshd disappear after a reboot?
• why did putting those 2 lines in the frontend not restore it?
• why did restoring it by hand work

I think /run is one of those directories that are populated at boot time? So why did it not populate properly?
Maybe because I did not enable sshd?

No it is not that - I rebooted with it enabled. /run/sshd still disappears after a boot.
and
This time restoring /run/sshd manually did not fix the issue?.. ie it will not start with 66 start… I can still start it by hand.

[ProwlerGr]

I have the impression that /run gets mounted on ramdisk (wiped at reboot)
Your updated frontend works fine for me (rebooted twice & sshd comes up normally)

@nevj

foreground { mkdir -p /run/sshd

This line is missing a closing bracket in your frontend, I’m thinking this might be the reason its failing for you…

Also remember after changing the frontend:

sudo 66 remove sshd
sudo 66 parse sshd
sudo 66 enable sshd
sudo 66 start sshd

antix@antix-init-diversity:~
$ cat /etc/66/service/sshd
[main]
@type = classic
@version = 0.0.2
@description = “ssh daemon”
@user = ( root )
[start]
@execute = (
foreground { mkdir -p /run/sshd }
foreground { chmod 0755 /run/sshd }
foreground { exec ssh-keygen -A }
/usr/sbin/sshd -e -D
)

antix@antix-init-diversity:~
$ sudo 66 -z tree status -go name,contents
Name : boot
Contents :
├─boot@system:system-hostname (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-run (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-tmp (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:populate-sys (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-pts (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-shm (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:populate-dev (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-cgroups (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:system-sysctl (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:udevd-log (pid=362, state=Enabled, type=classic, tree=boot)
├─boot@system:tty-earlier@tty12 (pid=271, state=Enabled, type=classic, tree=boot)
├─boot@system (pid=up, state=Enabled, type=module, tree=boot)
├─boot@system:populate-run (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:populate-tmp (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-branch (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:system-hwclock (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:system-random (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:udevd (pid=463, state=Enabled, type=classic, tree=boot)
├─boot@system:udevadm (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:system-fontnkey (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:system-fsck (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:system-branch (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-rw (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:local-time (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:local-loop (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:mount-swap (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:local-sethostname (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:local-authfiles (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:local-dmesg (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:local-branch (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:runtime-branch (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:canopy (pid=up, state=Enabled, type=oneshot, tree=boot)
├─boot@system:tty-rc@tty2 (pid=1100, state=Enabled, type=classic, tree=boot)
└─boot@system:tty-rc@tty1 (pid=1101, state=Enabled, type=classic, tree=boot)

Name : global
Contents :
├─dbus-log (pid=1034, state=Enabled, type=classic, tree=global)
├─seatd-log (pid=1038, state=Enabled, type=classic, tree=global)
├─slimski-log (pid=1031, state=Enabled, type=classic, tree=global)
├─connmand-log (pid=1042, state=Enabled, type=classic, tree=global)
├─nmbd-log (pid=1045, state=Enabled, type=classic, tree=global)
├─smbd-log (pid=1046, state=Enabled, type=classic, tree=global)
├─ufw (pid=up, state=Enabled, type=oneshot, tree=global)
├─bluetoothd-log (pid=1048, state=Enabled, type=classic, tree=global)
├─elogind-log (pid=0, state=Disabled, type=classic, tree=global)
├─sshd-log (pid=1049, state=Enabled, type=classic, tree=global)
├─dbus (pid=1058, state=Enabled, type=classic, tree=global)
├─seatd (pid=1061, state=Enabled, type=classic, tree=global)
├─nmbd (pid=1075, state=Enabled, type=classic, tree=global)
├─smbd (pid=1064, state=Enabled, type=classic, tree=global)
├─sshd (pid=1079, state=Enabled, type=classic, tree=global)
├─connmand (pid=1090, state=Enabled, type=classic, tree=global)
├─bluetoothd (pid=1089, state=Enabled, type=classic, tree=global)
├─elogind (pid=0, state=Disabled, type=classic, tree=global)
└─slimski (pid=1077, state=Enabled, type=classic, tree=global)

antix@antix-init-diversity:~
$

[nevj (Neville Jackson)]

@ProwlerGr

@nevj

foreground { mkdir -p /run/sshd

This line is missing a closing bracket in your frontend

OK my mistake there

Also remember after changing the frontend:

    sudo 66 remove sshd
    sudo 66 parse sshd
    sudo 66 enable sshd
    sudo 66 start sshd

I did not know about remove… and I guess that is why when I looked
at the script in /var/lib/66/system/service/svc/sshd it was an old version.

Do I really have to do enable before start ???
That does not make sense to me
enable means set it up for autostart at boot
start means start instantly
so why would start need enable?

Now I have a side issue. Tried another reboot and it would not boot…
hung at Stage2. I am doing a fresh install.

[ProwlerGr]

@nevj

I did not know about remove… and I guess that is why when I looked
at the script in /var/lib/66/system/service/svc/sshd it was an old version.

Do I really have to do enable before start ???
That does not make sense to me
enable means set it up for autostart at boot
start means start instantly
so why would start need enable?

Enable means add it to autostart at boot - it is optional not needed
Start means start immediately

@nevj

Now I have a side issue. Tried another reboot and it would not boot…
hung at Stage2. I am doing a fresh install.

Before attempting to reinstall
66 has an earlygetty console

Alt+ctrl+F12 would bring it up (Another great thing about 66)

[nevj (Neville Jackson)]

@ProwlerGr

Before attempting to reinstall
66 has an earlygetty console

Alt+ctrl+F12 would bring it up (Another great thing about 66)

Too late I have reinstalled. Antix is a really easy fast install

Enable means add it to startup
What is ‘startup’?
Yes, as I thought enable is optional… I think it may be a good policy to not enable until I am sure it works to avoid boot hangups.

Tested Ctrl Alt F12. Whole screen went blank,and it just echoes characters typed.
I dont think it works in a VM?

I will just redo sshd in the fresh install and see if the issue reappears.

[nevj (Neville Jackson)]

@nevj

Also remember after changing the frontend:

    sudo 66 remove sshd
    sudo 66 parse sshd
    sudo 66 enable sshd
    sudo 66 start sshd

Complete redo from scratchj

fix the frontend for sshd
$ cat /etc/66/service/sshd
[main]
@type=classic
@description="sshd daemon"
@version=0.0.2
@user=(root)

[start]
@execute=(
        foreground { mkdir -p /run/sshd }
	foreground { chmod 0755 /run/sshd }
	foreground { exec ssh-keygen -A }
    /usr/sbin/sshd -D -e 
)

root@antix-init-diversity:~# 66 parse sshd
parse: info: Parsed successfully: sshd-log at tree: global
parse: info: Parsed successfully: sshd at tree: global
parse: info: Parsed successfully: elogind-log at tree: global
parse: info: Parsed successfully: elogind at tree: global
66 sroot@antix-init-diversity:~# 66 start sshd
start: info: Initialized successfully: sshd
start: info: Initialized successfully: sshd-log
signal: info: Successfully started service: sshd-log
signal: info: Successfully started service: sshd
root@antix-init-diversity:~# ps ax | grep ssh
 2991 ?        S      0:00 s6-supervise sshd-log
 2992 ?        S      0:00 s6-supervise sshd
 2999 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/sshd
 3003 ?        Ss     0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups

So that works. The directory /run/sshd appears, so the script is making it.

Now it should survive a boot

cold bootoot@antix-init-diversity:/home/nevj# 66 start sshd
start: info: Initialized successfully: sshd
start: info: Initialized successfully: sshd-log
signal: info: Successfully started service: sshd-log
signal: info: Successfully started service: sshd
root@antix-init-diversity:/home/nevj# ps ax | grep ssh
 1366 ?        S      0:00 s6-supervise sshd-log
 1367 ?        S      0:00 s6-supervise sshd
 1374 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/sshd
 1378 ?        Ss     0:00 sshd: /usr/sbin/sshd -D -e [listener] 0 of 10-100 startups

Correct… all I need to do is 66 start sshd , because it is already parsed. If I enabled it , I would not need the start.

Now lets recap…

If , before it is parsed I change /etc/66/service/sshd ( the ‘frontend’) I need

66 parse sshd
66 start sshd

If, when sshd has already been parsed I change /etc/66/service/sshd
I need

66 remove sshd
66 parse sshd
66 start sshd

and if I reboot I need

66 start sshd

If I want it to start auotomatically I need

66 enable sshd

do that once only.
Correct?..
Then we can move on.

[ProwlerGr]

Yes.
Need to keep in mind that the frontend files are only used to create the services, they aren’t used directly by 66.
That is why you need to run the ‘66 remove’ command as it removes the service files, which you then recreate with the ‘66 parse’ command

[nevj (Neville Jackson)]

@ProwlerGr

That is why you need to run the ‘66 remove’ command as it removes the service files, which you then recreate with the ‘66 parse’ command

So 66 does not really have a database… there are just service files?

[ProwlerGr]

@nevj

So 66 does not really have a database… there are just service files?

The last paragraph states that the resolve files that are created after a parse are CDB databases.

https://web.obarun.org/software/66/latest/66-deeper.html

This would be the reason that the 66 author in no uncertain terms states to not modify/copy/copy any 66 files manually (with only exception being frontend files).
The 66 commands (as I understand it) automatically invoke various compilations when called.

[nevj (Neville Jackson)]

@ProwlerGr

resolve files that are created after a parse are CDB databases

OK it does use a database. CDB comes from Bernstein, the
author of daemontools.

I can see why the accessible parts of 66’s filesystem should not be touched… with few exceptions.
Maybe the whole thing needs to be hidden or protected?
It means that 66 needs to be bullet-proof… noone could ever fix a problem by hand intervention… I suppose that applies to systemd too?

With hindsight , I know why my system hung on boot… I enabled a daemon with a faulty frontend that had been parsed. … how would I get out of that without a reinstall?
There needs to be a way. I cant boot another linux and edit files… I need to use 66 commands to fix it… I need a way of
booting into the s6-66 system without starting the services?

[ProwlerGr]

@nevj

There needs to be a way. I cant boot another linux and edit files… I need to use 66 commands to fix it… I need a way of
booting into the s6-66 system without starting the services?

As I before-mentioned, 66 provides an early-getty console (Alt-Ctrl-F12)
It saved me literally dozens of times when I was trying to get 66 booting properly with antiX.

Once you open an early-getty session you’d run

mount -o remount,rw /

Which would give you read-write access to the system & take it from there.

You could also potentially use the “traditional” grub recovery method (using a shell as init)

[ProwlerGr]

Here is the thread that gives insight to all the history of the effort made to get s6 working on antiX.

https://www.antixforum.com/forums/topic/is-antix-going-to-get-s6-init-manager/page/16/#post-128564

Also FYI, I was using a virtualbox installation & can confirm that the earlygetty (Alt-Ctrl-F12) console worked very reliably at every occassion my boot would hang at either Stage1 or Stage2.

[nevj (Neville Jackson)]

@ProwlerGr

As I before-mentioned, 66 provides an early-getty console (Alt-Ctrl-F12)
It saved me literally dozens of times when I was trying to get 66 booting properly with antiX.

Once you open an early-getty session you’d run

mount -o remount,rw /

Which would give you read-write access to the system & take it from there.

All I get with virt-manager is a blank screen, no prompt.
If I type it echoes the characters typed, but does nothing?
I tried gnome-boxes, it is the same.
The getty process is running for tty12

$ ps ax | grep get
  299 tty12    Ss+    0:00 agetty -J 38400 tty12
  938 tty1     Ss+    0:00 agetty -J 38400 tty1
  939 tty2     Ss+    0:00 agetty -J 38400 tty2

But,
the grub method works
I edited the grub menu item for antix-s6-66, changed init to /usr/bin/sh, did Ctrl X, and it booted into a root shell.
Then did mount -o remount.rw / and I had the root filesystem and
it could find which 66 .
So that would work for me

I think I shall put the virt-manager issue with Ctrl Alt F12 on the shelf.

Thanks.

[nevj (Neville Jackson)]

@ProwlerGr

thread that gives insight to all the history of the effort made to get s6 working on antiX.

That is valuable background
Thanks

I found a useful (for me) tutorial on execline

Getting started with execline scripting

You probably dont need it.

[nevj (Neville Jackson)]

@nevj

Cor

@ProwlerGr ,
The sshd exercise is completed. I now know how to start a sdaemon which is already installed and has a frontend script. We even dealt with errors in the frontend, and after getting a working frontend I was able to ‘enable’ sshd so it starts at boot. .

Now, next exercise, lets look at starting a service which is not installed, and for which the package installed does not supply a ‘frontend’.
As with s6-rc, we can use vsftpd.

1. Install vsftpd using the Antix Package Installer… it is a nice simple gui package manager, similar to the one in MX. It may even BE the one in MX.

2. Check for a frontend

$ ls /etc/66/service
bluetoothd   dhcpcd          seatd    sshd.hide.orig  wpa-supplicant
boot@system  elogind         slimski  ubus            xrdp
connmand     NetworkManager  smbd     ufw             xrdp-sesman
dbus         nmbd            sshd     user

There is no frontend for vsftpd.

3. Write a frontend

[main]
@type=classic
@description="vsftpd daemon"
@version=0.0.1
@user=(root)

[start]
@execute=(
    /usr/sbin/vsftpd
)

We start with the simplest possible frontend

3. Try to parse it and start it

root@antix-init-diversity:~# 66 -v 4 parse vsftpd
parse: (parse_frontend(): 126): tracing: parse service: /etc/66/service/vsftpd
parse: (parse_frontend(): 141): tracing: read frontend service at: /etc/66/service/vsftpd
parse: (parse_split_from_section(): 76): tracing: parsing section: main
parse: (parse_split_from_section(): 161): tracing: parsing key: @type
parse: (parse_store_g(): 25): tracing: storing key: @type
...... lots of traces.... seems to work... no error

root@antix-init-diversity:~# 66 -v 4 start vsftpd
start: (debug_flag(): 74): tracing: requested flags to build the graph:  toparse topropagate wantup
.....  lots of output
signal: (announce(): 216): tracing: delete down file: /run/66/scandir/0/vsftpd/down
signal: (announce(): 230): info: Successfully started service: vsftpd

It says successfully started, lets see

root@antix-init-diversity:~# ps ax | grep ftp
 4668 ?        S      0:00 s6-supervise vsftpd-log
 4669 ?        S      0:00 s6-supervise vsftpd
 4680 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/vsftpd
 4684 ?        Ss     0:00 /usr/sbin/vsftpd
 4880 pts/1    S+     0:00 grep ftp

Yep. It even started the s6-log process, without me telling it anything about logs in the frontend.
So, compare this to s6-rc. The s6-rc effort required two execline scripts, one for vsftpd-srv and one for vsftpd-log. Here with s6-66 all I needed to write was one trivial frontend… admittedly vsftpd is a very simple daemon with no special requirements. I am getting to like s6-66.

4. Lets see if what I have done so far survives a boot?

reboot
root@antix-init-diversity:/home/nevj# ps ax | grep ftp
 1601 pts/1    S+     0:00 grep ftp
root@antix-init-diversity:/home/nevj# 66 start vsftpd
start: info: Initialized successfully: vsftpd
start: info: Initialized successfully: vsftpd-log
signal: info: Successfully started service: vsftpd-log
signal: info: Successfully started service: vsftpd
root@antix-init-diversity:/home/nevj# ps ax | grep ftp
 1636 ?        S      0:00 s6-supervise vsftpd-log
 1637 ?        S      0:00 s6-supervise vsftpd
 1648 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/vsftpd
 1654 ?        Ss     0:00 /usr/sbin/vsftpd
 1676 pts/1    S+     0:00 grep ftp

So it is not running after boot, because I did not enable it, so I use '66 start vsftpd` and it runs, along with the logging and supervision.

End of vsftpd exercise. Installing and starting a simple daemon is easily accomplished.

[nevj (Neville Jackson)]

Lets look at the execline scripts that were manufactured by 66 from our frontend.

Here is the frontend again ( in /etc/66/service/vsftpd)

[main]
@type=classic
@description="vsftpd daemon"
@version=0.0.1
@user=(root)

[start]
@execute=(
    /usr/sbin/vsftpd
)

And if we now look in /var/lib/66/system/service/svc/vsftpd we see

root@antix-init-diversity:/var/lib/66/system/service/svc/vsftpd# ls
max-death-tally  run  run.user	state

The 2 files run and run.usr containthe execline scripts

root@antix-init-diversity:/var/lib/66/system/service/svc/vsftpd# cat run
#!/usr/lib/execline/bin/execlineb -P
/usr/lib/execline/bin/fdmove 1 0
/usr/bin/s6-fdholder-retrieve /run/66/scandir/0/fdholder/s "pipe:66-w-vsftpd-log"
/usr/lib/execline/bin/fdswap 0 1
./run.user
root@antix-init-diversity:/var/lib/66/system/service/svc/vsftpd# cat run.user
#!/usr/lib/execline/bin/execlineb -P
/usr/lib/execline/bin/fdmove -c 2 1

    /usr/sbin/vsftpd

but, that is not all, there is also

/var/lib/66/system/service/svc/vsftpd-log# ls
notification-fd  run  run.user	state

root@antix-init-diversity:/var/lib/66/system/service/svc/vsftpd-log# cat run
#!/usr/lib/execline/bin/execlineb -P
s6-fdholder-retrieve /run/66/scandir/0/fdholder/s "pipe:66-r-vsftpd-log"
./run.user
root@antix-init-diversity:/var/lib/66/system/service/svc/vsftpd-log# cat run.user
#!/usr/lib/execline/bin/execlineb -P
/usr/lib/execline/bin/fdmove -c 2 1
/usr/bin/s6-setuidgid root
/usr/bin/s6-log -d3 n3 s1000000 /var/log/66/vsftpd

So 4 execline scripts have been created from the one simple frontend.
That is quite impressive.
All the technical details of execline scripting have been hidden from the user.

Issue:
The only documentation I could find on writing frontend files foe 66 is here
https://web.obarun.org/software/66/latest/66-frontend.html
it lists a minimal template, and a prototype containing every possible entry.