Version v0.14 of the documentation is no longer actively maintained. The site that you are currently viewing is an archived snapshot. For up-to-date documentation, see the latest version.


Viewing logs

Kernel messages can be retrieved with talosctl dmesg command:

$ talosctl -n dmesg kern:    info: [2021-11-10T10:09:37.662764956Z]: Command line: init_on_alloc=1 slab_nomerge pti=on consoleblank=0 nvme_core.io_timeout=4294967295 random.trust_cpu=on printk.devkmsg=on ima_template=ima-ng ima_appraise=fix ima_hash=sha512 console=ttyS0 reboot=k panic=1 talos.shutdown=halt talos.platform=metal talos.config=

Service logs can be retrieved with talosctl logs command:

$ talosctl -n services

NODE         SERVICE      STATE     HEALTH   LAST CHANGE   LAST EVENT   apid         Running   OK       19m27s ago    Health check successful   containerd   Running   OK       19m29s ago    Health check successful   cri          Running   OK       19m27s ago    Health check successful   etcd         Running   OK       19m22s ago    Health check successful   kubelet      Running   OK       19m20s ago    Health check successful   machined     Running   ?        19m30s ago    Service started as goroutine   trustd       Running   OK       19m27s ago    Health check successful   udevd        Running   OK       19m28s ago    Health check successful

$ talosctl -n logs machined [talos] task setupLogger (1/1): done, 106.109µs [talos] phase logger (1/7): done, 564.476µs

Container logs for Kubernetes pods can be retrieved with talosctl logs -k command:

$ talosctl -n containers -k
NODE         NAMESPACE   ID                                                 IMAGE                                                         PID    STATUS      kube-system/kube-flannel-dk6d5                                                     1329   SANDBOX_READY      └─ kube-system/kube-flannel-dk6d5:install-cni   0      CONTAINER_EXITED      └─ kube-system/kube-flannel-dk6d5:install-config                                0      CONTAINER_EXITED      └─ kube-system/kube-flannel-dk6d5:kube-flannel                                1610   CONTAINER_RUNNING      kube-system/kube-proxy-gfkqj                                                       1311   SANDBOX_READY      └─ kube-system/kube-proxy-gfkqj:kube-proxy                                 1379   CONTAINER_RUNNING

$ talosctl -n logs -k kube-system/kube-proxy-gfkqj:kube-proxy 2021-11-30T19:13:20.567825192Z stderr F I1130 19:13:20.567737       1 server_others.go:138] "Detected node IP" address="" 2021-11-30T19:13:20.599684397Z stderr F I1130 19:13:20.599613       1 server_others.go:206] "Using iptables Proxier"

Sending logs

Service logs

You can enable logs sendings in machine configuration:

      - endpoint: "udp://"
        format: "json_lines"
      - endpoint: "tcp://host:5044/"
        format: "json_lines"

Several destinations can be specified. Supported protocols are UDP and TCP. The only currently supported format is json_lines:

  "msg": "[talos] apply config request: immediate true, on reboot false",
  "talos-level": "info",
  "talos-service": "machined",
  "talos-time": "2021-11-10T10:48:49.294858021Z"

Messages are newline-separated when sent over TCP. Over UDP messages are sent with one message per packet. msg, talos-level, talos-service, and talos-time fields are always present; there may be additional fields.

Kernel logs

Kernel log delivery can be enabled with the talos.logging.kernel kernel command line argument, which can be specified in the .machine.installer.extraKernelArgs:

      - talos.logging.kernel=tcp://host:5044/

Kernel log destination is specified in the same way as service log endpoint. The only supported format is json_lines.

Sample message:

  "clock":6252819, // time relative to the kernel boot time
  "msg":"[talos] task startAllServices (1/1): waiting for 6 services\n",
  "talos-level":"warn", // Talos-translated `priority` into common logging level
  "talos-time":"2021-11-26T16:53:21.3258698Z" // Talos-translated `clock` using current time

extraKernelArgs in the machine configuration are only applied on Talos upgrades, not just by applying the config. (Upgrading to the same version is fine).

Filebeat example

To forward logs to other Log collection services, one way to do this is sending them to a Filebeat running in the cluster itself (in the host network), which takes care of forwarding it to other endpoints (and the necessary transformations).

If Elastic Cloud on Kubernetes is being used, the following Beat (custom resource) configuration might be helpful:

kind: Beat
  name: talos
  type: filebeat
  version: 7.15.1
    name: talos
      - type: "udp"
        host: ""
          - decode_json_fields:
              fields: ["message"]
              target: ""
          - timestamp:
              field: "talos-time"
                - "2006-01-02T15:04:05.999999999Z07:00"
          - drop_fields:
              fields: ["message", "talos-time"]
          - rename:
                - from: "msg"
                  to: "message"

        maxUnavailable: 100%
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
          runAsUser: 0
          - name: filebeat
              - protocol: UDP
                containerPort: 12345
                hostPort: 12345

The input configuration ensures that messages and timestamps are extracted properly. Refer to the Filebeat documentation on how to forward logs to other outputs.

Also note the hostNetwork: true in the daemonSet configuration.

This ensures filebeat uses the host network, and listens on (UDP) on every machine, which can then be specified as a logging endpoint in the machine configuration.