docs: record the telega docker pin and tdlib version mismatch

MELPA's 2026-08-18 telega build raised telega-tdlib-min-version to 1.8.66. The pinned zevlg/telega-server image ships 1.8.64, so telega kills its own server on every launch and nothing reports it.

The note carries the measurements and why the armed death alert wasn't seen, which is still unmeasured. It also records why zevlg/telega-server:latest isn't a fix (zevlg/telega.el#596: missing libglycin), the derived image I verified, and the pinning decision I haven't made yet.

1 files changed, 232 insertions, 0 deletions

diff --git a/docs/design/2026-08-23-telega-docker-pin-tdlib-mismatch.org b/docs/design/2026-08-23-telega-docker-pin-tdlib-mismatch.org
new file mode 100644
index 00000000..919c84b8
--- /dev/null
+++ b/docs/design/2026-08-23-telega-docker-pin-tdlib-mismatch.org
@@ -0,0 +1,232 @@
+#+TITLE: telega-server is dead on velox: the docker image pin went stale when the package updated
+#+AUTHOR: Craig Jennings
+#+DATE: 2026-08-23
+
+* Summary
+
+Telegram has been an unscannable channel on velox since 2026-08-19. Two triage
+sweeps (08-21 and 08-23) recorded it as a scan failure and both blamed a missing
+docker image. That diagnosis was wrong. The real cause is a version mismatch
+between =modules/telega-config.el='s digest pin and the telega package MELPA
+installed on 2026-08-18. It is reproducible, it is silent, and ratio is one
+package upgrade from the same failure.
+
+Everything below was measured on velox on 2026-08-23, with ratio checked over
+tailscale for comparison.
+
+* What actually happens
+
+=telega-server= starts fine. The container runs, tdlib initialises, and telega.el
+completes a handshake with it: the server log shows five =setOption= requests
+arriving from telega.el (language, =use_storage_optimizer=, =ignore_file_names=).
+About eleven milliseconds later telega.el sends =close= and the client unwinds
+cleanly: requests aborted, =authorizationStateClosing=,
+=authorizationStateClosed=, =Td::hangup=, =Stop Td=.
+
+The trigger is in =telega-tdlib-events.el= around line 1306. When tdlib reports
+its =:version= option, telega compares it against =telega-tdlib-min-version= and,
+if the server is older, does this:
+
+#+begin_src emacs-lisp
+(warn version-error-msg)
+;; Finally stop processing events from telega-server
+(telega-server-kill)
+(error version-error-msg)
+#+end_src
+
+The numbers:
+
+| where                                              | value  |
+|----------------------------------------------------+--------|
+| tdlib in the pinned image                          | 1.8.64 |
+| =telega-tdlib-min-version= in telega-20260817.621  | 1.8.66 |
+
+1.8.64 < 1.8.66, so telega kills its own server on every launch.
+
+* The timeline is exact
+
+- =elpa/telega-20260817.621/= is dated 2026-08-18 11:14. The package updated,
+  and that build raised =telega-tdlib-min-version= from 1.8.64 to 1.8.66.
+- =~/.telega/telega-server.log= holds at least six launch attempts across
+  2026-08-19 15:20, 08-20 10:50, 08-21 15:10 and 08-23 10:50 (the count grows by
+  one each time I reproduce it). Every one reaches
+  =authorizationStateWaitTdlibParameters= and then closes. Not one ever reaches
+  =WaitPhoneNumber=, let alone Ready.
+- The first failure is the day after the package update. Nothing else changed.
+
+The digest pin is
+=zevlg/telega-server@sha256:a4b88e029ba381eca7c37c9618c9e3ad73aa9db2097fe07a0c6684d40d32b84e=,
+an image built 2026-06-05. It was correct for the July package and is wrong for
+this one.
+
+* Two things I got wrong first, worth recording
+
+The image is present. I reported it missing on 08-21 and again on 08-23, on the
+strength of =docker images= showing only the slack-mcp container. That output
+lies here: the newer docker CLI's default listing (the =IMAGE / DISK USAGE /
+CONTENT SIZE= format) only shows tagged images, and an image pulled by digest has
+no =RepoTags=. =docker images -a --digests= shows it plainly:
+
+: zevlg/telega-server   <none>   sha256:a4b88e02...   db0e011ad225   2 months ago   312MB
+
+If a probe needs to know whether an image is on a machine, =docker image inspect
+<ref>= is the honest check. =docker images= isn't.
+
+The image also works. I ran the exact command telega builds
+(=telega-docker-run-cmd=) by hand. tdlib 1.8.64 came up, created a client, and sat
+at =authorizationStateWaitTdlibParameters= waiting for telega.el, which is
+correct behaviour. There's nothing wrong with the container.
+
+* Why nothing told me, and what I still don't know about that
+
+I first wrote this up as "the death alert is structurally unable to fire because
+=telega-server-kill= is orderly, so the exit status is zero." That was wrong, and
+the evidence that refutes it was already in front of me: the daemon's
+=*Messages*= holds =[125]telega-server: exited abnormally with code 125=, and
+=telega-server--sentinel= prints that line only from its non-zero-exit branch.
+=telega-server-kill= works by killing the process buffer, not by a clean close,
+so the process exits 125. =cj/--telega-server-death-p= returns t for any non-zero
+integer, and the =:after= advice on the sentinel then runs
+=notify fail "Telegram: telega-server died" ... --persist=.
+
+So the notification path was armed and should have paged me on every one of
+those launches. I didn't see one. I can't reconstruct why from here: the
+daemon's process table from those days is gone, so whether =notify= ran and I
+dismissed it, or the =notify= script was absent in the days after the 08-13
+rebuild, or the advice hadn't been installed in that daemon, is unmeasured.
+Worth a controlled reproduction with the daemon watched, rather than another
+guess.
+
+Two things did fail as described. The =(warn ...)= goes to =*Warnings*=, which I
+never look at in a daemon driven by =emacsclient=. And =(telega t)= still returns
+=started= regardless, so a caller checking the return value learns nothing; only
+=(telega-server-live-p)= a moment later tells the truth.
+
+* ratio is fine today and breaks on its next package upgrade
+
+|                                                | velox               | ratio                |
+|------------------------------------------------+---------------------+----------------------|
+| telega package                                 | telega-20260817.621 | telega-20260706.2147 |
+| =telega-tdlib-min-version=                     | 1.8.66              | 1.8.64               |
+| pinned image (a4b88e, tdlib 1.8.64)            | present             | present              |
+| =zevlg/telega-server:latest= (aa7e79c4)        | absent (pulled later, see below) | present |
+| =~/.telega/td.binlog=                          | absent              | 1.5 MB, written 08-16 |
+| =~/.telega/db.sqlite=                          | absent              | 14.7 MB              |
+
+ratio's package still wants 1.8.64, which is exactly what the pinned image
+carries, which is why Telegram still works there. It's one =package-upgrade
+telega= away from the identical silent failure. Worth fixing before that happens
+rather than after.
+
+* Second, separate problem on velox: there is no Telegram session
+
+=~/.telega/= on velox holds only an empty =cache/= and =temp/=, both dated
+2026-08-19 15:26, the first failed attempt. No =td.binlog=, no =db.sqlite=. The
+account state was lost in the 2026-08-13 rebuild and never restored (the same
+rebuild that wiped the Signal identity, which I recovered from the pre-reinstall
+backup).
+
+So fixing the image pin gets telega to start on velox. It will then sit at
+=WaitPhoneNumber= and need interactive auth (phone number plus verification code)
+inside =M-x telega=. That can't be done headlessly. The pre-reinstall backup is
+worth checking for =~/.telega= before re-authing from scratch, since restoring
+the binlog would keep the existing session rather than minting a new device.
+
+* The obvious fix doesn't work
+
+The mechanical fix would be to move =cj/telega-docker-image= to a digest whose
+tdlib is >= 1.8.66. The obvious candidate is =zevlg/telega-server:latest=, which
+ratio already holds at
+=sha256:aa7e79c4e9587c84065782df9399653fb1a967078b08ec1ded77738d4c2d70ad=.
+
+I read the tdlib version straight out of the shared library in both images on
+ratio, where both are present:
+
+| image                  | libtdjson             |
+|------------------------+-----------------------|
+| =a4b88e02= (pinned)    | =libtdjson.so.1.8.64= |
+| =aa7e79c4= (=:latest=) | =libtdjson.so.1.8.66= |
+
+The version floor is satisfied by =:latest=, but the image is broken:
+=docker run --rm zevlg/telega-server:latest telega-server -h= exits 127 with 32
+lines of linker failure, beginning
+
+: Error loading shared library libglycin-2.so.0: No such file or directory (needed by /usr/lib/libgdk_pixbuf-2.0.so.0)
+
+It reproduced identically on velox after I pulled the image there, so it's the
+image and not one machine. My first read blamed the failure on having overridden
+the container entrypoint. That's not it: the image has no entrypoint
+(=ENTRYPOINT=null=, =CMD=["/bin/bash"]=), and the same invocation against the
+pinned image runs and prints =Version 1.2.0, with appindicator, with zlib=.
+
+** It's a known, open upstream bug
+
+[[https://github.com/zevlg/telega.el/issues/596][zevlg/telega.el issue #596]],
+"docker image zevlg/telega-server:latest missing libglycin (-2.so.0)", opened
+2026-08-11 by akovalenko. Still open, no maintainer response, no official fix.
+
+Docker Hub carries only three tags for =zevlg/telega-server=: =latest= (pushed
+2026-07-23, digest =aa7e79c4=), =arm64= (2025-02-03) and =1.8.0= (2021-12-31).
+So there's no =1.8.66= tag to pin to instead, and "pin to latest" and "pin to the
+broken image" are the same action.
+
+** The workaround works, verified locally
+
+The two-line derived image from issue #596:
+
+#+begin_src dockerfile
+FROM zevlg/telega-server:latest
+RUN apk add --no-cache libglycin
+#+end_src
+
+Built on velox as =telega-server-glycin:test=:
+
+- =apk add libglycin= resolves cleanly (150 packages, 155.6 MiB).
+- =telega-server -h= prints =Version 1.2.0, with appindicator, with zlib=. It runs.
+- =/usr/lib/libtdjson.so.1.8.66=, satisfying telega-20260817.621's floor.
+
+* What remains a decision rather than a measurement
+
+1. A locally-built image has no =RepoDigest=, so =cj/telega-docker-image= would
+   degrade from a digest pin to a tag reference, losing the guarantee that pin
+   exists to give.
+2. It's fleet-local. ratio needs the same image built there (or a registry to
+   pull from), or it breaks on its next =package-upgrade telega=.
+3. The alternative is pinning the telega package back to =20260706.2147=. That
+   keeps the working upstream image and the digest pin, and stops the
+   auto-upgrade that caused this, at the cost of freezing telega until #596 is
+   fixed.
+4. Either way velox has no Telegram session, so it lands at =WaitPhoneNumber=
+   and needs interactive auth that can't be done headlessly.
+
+I haven't decided yet. Two related pieces of hardening are worth doing whichever
+way it goes:
+
+- The pin's own docstring already ties it to "this telega's
+  =telega-tdlib-min-version=", so the coupling was understood, but nothing
+  enforces it and the package updates on its own schedule. A launch-time
+  assertion comparing the running server's reported tdlib version against
+  =telega-tdlib-min-version=, reported through =notify= rather than =warn=,
+  catches the whole class. =tests/test-telega-config--docker-pin.el= is the
+  natural home for a test that the pin and the installed package's floor agree.
+- A controlled reproduction of the silent death with the daemon watched, to
+  find out why the armed notification wasn't seen (previous section).
+
+* Reproducing it
+
+#+begin_src sh
+# the image is there, despite what `docker images` shows
+docker images -a --digests | grep telega
+docker image inspect zevlg/telega-server@sha256:a4b88e029ba381eca7c37c9618c9e3ad73aa9db2097fe07a0c6684d40d32b84e >/dev/null && echo present
+
+# the version floor vs what the image ships
+grep -E 'defconst telega-tdlib-(min|max)-version' ~/.emacs.d/elpa/telega-*/telega.el
+
+# repeated identical failures, no Ready among them
+grep -c 'authorizationStateClosed' ~/.telega/telega-server.log
+grep 'Receive request 6: close' ~/.telega/telega-server.log
+
+# the broken upstream image, and the derived one that works
+docker run --rm zevlg/telega-server:latest telega-server -h; echo "exit $?"
+docker run --rm telega-server-glycin:test telega-server -h; echo "exit $?"
+#+end_src