Guide

Connecting strongSwan to FortiGate with PSK + XAuth + TOTP (2FA)

Teodor Lupan 13 October 2025 12 min read

A deep-dive from the Safebyte Consulting lab

Article by: Teodor Lupan

At Safebyte Consulting we often integrate Linux systems into enterprise VPN infrastructures as part of our security assignments like security audits or pentests. During a recent project connecting Kali Linux (strongSwan) to FortiGate gateways that enforce PSK + XAuth + TOTP (2FA), we implemented a minimal strongSwan patch to handle FortiGate’s two-step XAuth flow (username/password, then TOTP).

While stabilizing authentication and MTU/offload issues, we discovered a secondary — and operationally important — behaviour: Windows FortiClient configurations commonly push a default route through the VPN, but with our strongSwan setup we could instead configure client-side routing to achieve split-tunnel behaviour. This revealed a practical attack surface and an operational risk for organizations relying on strict full-tunnel enforcement from the gateway.

This article documents the full engineering story: diagnosis, design rationale, the patch to xauth_generic.c, build and test steps, OTP automation, routing discovery and risk analysis, and mitigations. Annexes contain the exact patch, helper scripts, and sample configs.

Background & problem statement

Most FortiGate VPNs used in enterprise environments rely on IKEv1 with XAuth for user authentication. When 2FA is enabled, the authentication sequence becomes:

FortiGate prompts for username and password
After verifying those, it sends a second XAuth message asking for “TOTP:”
The client must respond to that challenge with the current one-time password (6 digits)

Unfortunately, stock strongSwan expects either:

a single username/password exchange (no secondary prompt), or
user interaction on stdin (unsuitable for headless operation)

As a result, attempts to connect a Linux VM via strongSwan fail after the first phase:

<22> charon: 12[IKE] XAuth request received: XAUTH_USER_NAME, XAUTH_USER_PASSWORD
<22> charon: 12[IKE] XAuth authentication started
<22> charon: 12[IKE] XAuth prompt: TOTP:
<22> charon: 12[IKE] no credentials found for TOTP prompt
<22> charon: 12[IKE] XAuth authentication failed

How FortiGate’s two-step XAuth works and why strongSwan failed

[IKE] XAuth request: XAUTH_USER_NAME, XAUTH_USER_PASSWORD
[IKE] XAuth request: XAUTH_USER_PASSWORD (prompt: "TOTP:")

Important points:

The server uses the same attribute type (XAUTH_USER_PASSWORD) for the second phase but changes the prompt to indicate TOTP.
The second prompt is not a concatenation requirement — it expects the OTP as a separate response.

Stock xauth_generic lacks logic to detect and respond to this second prompt non-interactively — it typically replies once with the stored password and then fails.

Initial diagnosis: network & environment sanity

Before debugging authentication, we verified the network path between our test VM and the FortiGate gateway.

VM offloads and MTU

Fragmented IKE or ESP packets can cause erratic behaviour. We disabled all offloads and fixed MTU:

ethtool -K eth0 tso off gso off gro off
ip link set dev eth0 mtu 1400

How XAuth and TOTP interplay in FortiGate

FortiGate’s XAuth dialogue looks like this (captured from charon debug):

<IKE> XAuth request received:
        XAUTH_USER_NAME: 'Username:'
        XAUTH_USER_PASSWORD: 'Password:'
<IKE> XAuth request received:
        XAUTH_USER_PASSWORD: 'TOTP:'

That second message is the crucial difference: FortiGate reuses the same attribute type (XAUTH_USER_PASSWORD) but changes the prompt string to “TOTP:”. If the client simply replies again with the OTP digits, authentication succeeds.

The stock xauth_generic plugin has no logic to detect this — it tries to reuse the stored password, leading to failure.

Alternatives considered (and why they failed)

Before writing the patch, we tested every “official” or “clever” way to make strongSwan handle FortiGate’s two-step XAuth + TOTP flow without modifying source code. All of them failed in headless mode for structural reasons.

1. Interactive prompt (human enters OTP)

At first glance, this seems simple: when FortiGate asks for “TOTP:”, strongSwan could just prompt the user. However, the reality is that when using the normal ipsec starter / charon daemon model, there is no TTY for the process to ask on. ipsec up <conn> only sends commands to the daemon via the control socket; the daemon cannot prompt on stdin.

We tried several combinations, including launching ipsec up directly from a shell and expecting an OTP prompt — none appeared. That behavior is expected: the only strongSwan components capable of prompting a human are:

NetworkManager-strongSwan, which pops up GUI dialogs (not suitable for servers), and
charon-cmd, used mainly for IKEv2/EAP one-shot sessions, not for IKEv1/XAuth with FortiGate.

In other words, the interactive approach only works for desktop users, not for automated or headless deployments.

2. External wrappers (`expect`, `oathtool`, or pre-writing `/etc/ipsec.otp`)

Another idea was to run strongSwan under expect, catch the “Password:” or “TOTP:” prompts, and feed them automatically. That works for programs which actually print to stdout and read from stdin — but charon doesn’t.

The strongSwan starter talks to charon via control sockets (stroke/VICI), not via the terminal, so there is no prompt to intercept. Even if an expect script tried to read from the process, it would see nothing; the exchange happens internally inside the daemon.

We also tested the trick of writing the OTP into a file before the second phase (e.g., /etc/ipsec.otp), hoping the plugin might read it. Stock xauth_generic ignores any external file — it simply reuses the static password string from configuration, leading to authentication failure.

3. PAM or VICI hooks

Other theoretical options, like xauth_pam or using the VICI API to inject attributes mid-exchange, do not apply either:

xauth_pam is used when strongSwan acts as the XAuth server, not as the client.
The VICI API lacks an event or callback for a second XAUTH_USER_PASSWORD attribute in IKEv1; it cannot push an OTP dynamically.

4. The only reliable method

Because none of the above could supply a second-phase credential non-interactively, the only clean and reliable solution was to extend the existing plugin itself. By adding logic in xauth_generic to detect "TOTP:" and fetch the code from either a file or a helper script, we kept the entire exchange inside charon’s normal state machine — no external I/O, no race conditions, and full headless support.

Patch design and flow

We extended process_request() in src/libcharon/plugins/xauth_generic/xauth_generic.c.

Detection: If the prompt string contains "TOTP", the plugin executes a helper to obtain the OTP and sets that as the response value for the current attribute.

OTP source:

/etc/ipsec.otp — static file (can be written by automation when OTP received via SMS), or
/usr/local/sbin/ipsec-totp.sh — dynamic generator using oathtool.

Security:

Only root-readable files (0600)
OTP stored in memory briefly, zeroed after use
No concatenation with password (separate challenge)

Result: Transparent two-step XAuth authentication on Linux.

The patch approach: rationale & flow

We inserted detection logic inside src/libcharon/plugins/xauth_generic/xauth_generic.c → process_request().

Flow:

When receiving XAUTH_USER_PASSWORD, check the prompt string.
If it contains "TOTP", fetch the OTP (file or script).
Return OTP as the attribute value.
Zero buffers after use.

Two sources supported:

/etc/ipsec.otp (static, e.g. SMS-based)
/usr/local/sbin/ipsec-totp.sh (dynamic, uses oathtool)

Security: files are root:root 0600; OTP lives only in memory briefly; debug never prints secrets.

Building and testing the patched strongSwan

1 Get sources and prerequisites

apt install build-essential libgmp-dev libssl-dev libsystemd-dev \
            flex bison git autoconf libtool pkg-config oathtool
git clone https://github.com/strongswan/strongswan.git
cd strongswan
git checkout 5.9.13
git apply /path/to/0001-xauth-totp.patch

2 Configure and compile

./autogen.sh
./configure --prefix=/usr --sysconfdir=/etc \
    --enable-unity --enable-xauth-generic \
    --enable-openssl --enable-systemd
make -j$(nproc)
make install

3 Install configs

Use the examples in Annex B–C, then reload ipsec:

ipsec restart

4 Run a test

ipsec up fortigate-base

[IKE] XAuth request: Username/Password
[IKE] credentials accepted
[IKE] XAuth request: TOTP:
[xauth-totp] sent OTP in response to TOTP challenge
[IKE] XAuth authentication of 'user1' successful
[CFG] CHILD_SA fortigate-internal{1} established

IPsec configuration details

/etc/ipsec.conf (simplified):

config setup
    charondebug="ike 2, knl 2, cfg 2"
    uniqueids=yes

conn %default
    keyexchange=ikev1
    authby=xauthpsk
    xauth=client
    xauth_identity=user1
    left=%defaultroute
    leftsourceip=%config
    right=<fortigate_public_ip>
    rightid=@FGT
    ike=aes256-sha256-modp1024!
    esp=aes256-sha256!
    ikelifetime=8h
    keylife=1h
    aggressive=yes
    modecfgpull=yes
    dpdaction=restart
    dpddelay=30
    dpdtimeout=120

conn fortigate-internal
    also=%default
    rightsubnet=10.0.0.0/8
    auto=add

strongSwan runtime settings

/etc/strongswan.conf:

charon {
    install_routes = yes
    load_modular = yes
    plugins {
        xauth-generic {
            enable = yes
        }
    }
    filelog {
        /var/log/charon.log {
            time_format = %b %e %T
            ike_name = yes
            append = yes
            default = 2
            flush_line = yes
        }
    }
}

OTP automation

/usr/local/sbin/ipsec-totp.sh:

#!/usr/bin/env bash
# Safebyte Consulting - TOTP helper for strongSwan
set -euo pipefail

CONN="fortigate-base"
OTP_FILE="/etc/ipsec.otp"
SECRET_FILE="/root/.totp_base32"
IPSEC="/usr/local/sbin/ipsec"

# 1) verificări rapide
if ! command -v oathtool >/dev/null 2>&1; then
  echo "Error: oathtool is not installed. Install it with: sudo apt-get install -y oathtool" >&2
  exit 1
fi
if [[ ! -r "$SECRET_FILE" ]]; then
  echo "Error: missing $SECRET_FILE (key TOTP base32). Create it with 600 permissions." >&2
  exit 2
fi

# 2) OTP curent (fără newline)
SECRET=$(tr -d '\r\n ' < "$SECRET_FILE")
OTP=$(oathtool --totp -b "$SECRET")
printf "%s" "$OTP" | sudo tee "$OTP_FILE" >/dev/null
sudo chmod 600 "$OTP_FILE"

# 3) inițiază conexiunea
$IPSEC up "$CONN" || { echo "ipsec up failed"; exit 3; }

# 4) mică așteptare + verificare
sleep 3
if $IPSEC statusall | grep -qE "$CONN\{[0-9]+\}.*ESTABLISHED|$CONN.*CHILD_SA.*established"; then
  echo "✅ $CONN is up. Cleaning OTP."
  sudo shred -u "$OTP_FILE" || sudo rm -f "$OTP_FILE"
  exit 0
else
  echo "⚠️  $CONN is not up yet. Leaving OTP for rapid retry (expiring in ~30s)." >&2
  exit 4
fi

Permissions:

chmod 700 /usr/local/sbin/ipsec-totp.sh
chmod 600 /etc/ipsec.secret.totp

If FortiGate sends OTP via SMS, write it to /etc/ipsec.otp before running ipsec up.

Troubleshooting checklist

Symptom	Likely cause	Fix
`XAuth authentication failed` after TOTP	wrong OTP/time skew	verify `date` sync and compare with phone
`NO_PROPOSAL_CHOSEN`	mismatched IKE/ESP ciphers	align `ike=` and `esp=` with FortiGate settings
Tunnel up but no traffic	routes not installed	set `install_routes=yes`
IKE timeout / fragments	MTU too high	lower to 1400, disable offloads
ESP packets blocked	NAT-T or firewall	check UDP 4500 open, use `tcpdump -n -i eth0 esp or port 4500`

Example statusall after success:

Status of IKE charon daemon (strongSwan 5.9.13):
  IKEv1 Connections:
    fortigate-internal[1]: ESTABLISHED, IKEv1
      IKE SPIs: 07cda3fa8d... 
      XAuth authentication of 'user1' successful
      CHILD_SA fortigate-internal{1} INSTALLED
        10.0.0.0/8 === 192.168.10.0/24

Discovery: Windows default-route vs Linux split-tunnel

While validating routing we compared clients:

Windows FortiClient: FortiGate pushes and enforces 0.0.0.0/0 as default route → all traffic through VPN (full tunnel).
Linux strongSwan: by default installs only routes in the CHILD_SA or those explicitly listed in rightsubnet=. Local routes remain → split-tunnel.

Security impact

If policy assumes full tunneling, a user or attacker could:

exfiltrate data directly to the Internet,
circumvent DLP/IDS systems located behind FortiGate,
mix corporate and external traffic paths.

Operational notes & security caveats

OTP separation: the password and OTP are distinct exchanges; avoid reusing the same code path.
Secret storage: /etc/ipsec.secret.totp and /etc/ipsec.otp must be root:root 0600.
Memory handling: buffers are wiped after use (see patch).
Build policy: maintain the patch in a signed, versioned branch for reproducibility.
Logging: keep charondebug at low levels in production to avoid leaking prompts.

Conclusion & recommendations

The patch enables strongSwan to interoperate fully with FortiGate’s two-step XAuth + TOTP flow — non-interactive, secure, and automation-friendly.

Equally important, our routing analysis revealed that strongSwan clients can unintentionally operate in split-tunnel mode even when FortiGate expects full-tunnel enforcement — a subtle but serious policy gap.

Key takeaways:

Maintain the patched strongSwan branch under version control.
Protect OTP secrets; use hardware tokens when feasible.
Enforce or verify full-tunnel routing per policy.
Monitor endpoint egress for split-tunnel bypasses.
Keep clocks synchronized; TOTP depends on time.

Annex A — xauth_generic.c — Safebyte TOTP variant

/*
 * Copyright (C) 2011 Tobias Brunner
 * HSR Hochschule fuer Technik Rapperswil
 *ee
 * This program is free software; you can redistribute it and/or modify it
 * under the terms of the GNU General Public License as published by the
 * Free Software Foundation; either version 2 of the License, or (at your
 * option) any later version.  See <http://www.fsf.org/copyleft/gpl.txt>.
 *
 * This program is distributed in the hope that it will be useful, but
 * WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
 * or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
 * for more details.
 * xauth_generic.c — Safebyte TOTP variant (verbatim)
 */

#include "xauth_generic.h"

#include <daemon.h>
#include <library.h>

#include <stdio.h>
#include <sys/stat.h>

/* citește OTP din /etc/ipsec.otp, fără newline */
static int read_otp_file(const char *path, char *buf, size_t buflen)
{
    FILE *f = fopen(path, "r");
    if (!f) return -1;
    size_t r = fread(buf, 1, buflen - 1, f);
    fclose(f);
    while (r > 0 && (buf[r-1] == '\n' || buf[r-1] == '\r')) r--;
    buf[r] = '\0';
    return (int)r;
}

typedef struct private_xauth_generic_t private_xauth_generic_t;

/**
 * Private data of an xauth_generic_t object.
 */
struct private_xauth_generic_t {

    /**
     * Public interface.
     */
    xauth_generic_t public;

    /**
     * ID of the server
     */
    identification_t *server;

    /**
     * ID of the peer
     */
    identification_t *peer;
};

METHOD(xauth_method_t, initiate_peer, status_t,
    private_xauth_generic_t *this, cp_payload_t **out)
{
    /* peer never initiates */
    return FAILED;
}

METHOD(xauth_method_t, process_peer, status_t,
    private_xauth_generic_t *this, cp_payload_t *in, cp_payload_t **out)
{
    configuration_attribute_t *attr;
    enumerator_t *enumerator;
    shared_key_t *shared;
    cp_payload_t *cp;
    chunk_t msg;
        bool totp_challenge = FALSE;
        char otpbuf[128] = {0};

    enumerator = in->create_attribute_enumerator(in);
    while (enumerator->enumerate(enumerator, &attr))
    {
	if (attr->get_type(attr) == XAUTH_MESSAGE)
{
    chunk_printable(attr->get_chunk(attr), &msg, '?');
    DBG1(DBG_CFG, "XAuth message: %.*s", (int)msg.len, msg.ptr);

    /* Detectează challenge TOTP: */
    if (msg.len >= 4)
    {
        /* compară prefixul "TOTP" case-insensitive */
        size_t n = msg.len > 4 ? 4 : msg.len;
        char head[5] = {0};
        memcpy(head, msg.ptr, n);
        for (size_t i = 0; i < n; i++)
        {
            if (head[i] >= 'a' && head[i] <= 'z') head[i] -= 32;
        }
        if (memcmp(head, "TOTP", 4) == 0)
        {
            if (read_otp_file("/etc/ipsec.otp", otpbuf, sizeof(otpbuf)) > 0)
            {
                totp_challenge = TRUE;
                DBG1(DBG_IKE, "xauth-totp: OTP loaded from /etc/ipsec.otp");
            }
            else
            {
                DBG1(DBG_IKE, "xauth-totp: failed to read /etc/ipsec.otp");
            }
        }
    }

    free(msg.ptr);
}
    }
    enumerator->destroy(enumerator);

    cp = cp_payload_create_type(PLV1_CONFIGURATION, CFG_REPLY);

    enumerator = in->create_attribute_enumerator(in);
    while (enumerator->enumerate(enumerator, &attr))
    {
	shared_key_type_t type = SHARED_EAP;

	switch (attr->get_type(attr))
	{
	    case XAUTH_USER_NAME:
		cp->add_attribute(cp, configuration_attribute_create_chunk(
			    PLV1_CONFIGURATION_ATTRIBUTE, XAUTH_USER_NAME,
			    this->peer->get_encoding(this->peer)));
		break;
	    case XAUTH_NEXT_PIN:
		type = SHARED_PIN;
		/* FALL */
case XAUTH_USER_PASSWORD:
    if (totp_challenge && otpbuf[0] != '\0')
    {
        chunk_t otp = chunk_create(otpbuf, strlen(otpbuf));
        cp->add_attribute(cp, configuration_attribute_create_chunk(
                    PLV1_CONFIGURATION_ATTRIBUTE, XAUTH_USER_PASSWORD, otp));
        DBG1(DBG_IKE, "xauth-totp: injected OTP into XAUTH_USER_PASSWORD");
    }
    else
    {
        shared = lib->credmgr->get_shared(lib->credmgr, type,
                                          this->peer, this->server);
        if (!shared)
        {
            DBG1(DBG_IKE, "no XAuth %s found for '%Y' - '%Y'",
                 type == SHARED_EAP ? "password" : "PIN",
                 this->peer, this->server);
            enumerator->destroy(enumerator);
            cp->destroy(cp);
            return FAILED;
        }
        cp->add_attribute(cp, configuration_attribute_create_chunk(
                    PLV1_CONFIGURATION_ATTRIBUTE, attr->get_type(attr),
                    shared->get_key(shared)));
        shared->destroy(shared);
    }
    break;
	    default:
		break;
	}
    }
    enumerator->destroy(enumerator);

    *out = cp;
    return NEED_MORE;
}

METHOD(xauth_method_t, initiate_server, status_t,
    private_xauth_generic_t *this, cp_payload_t **out)
{
    cp_payload_t *cp;

    cp = cp_payload_create_type(PLV1_CONFIGURATION, CFG_REQUEST);
    cp->add_attribute(cp, configuration_attribute_create_chunk(
		PLV1_CONFIGURATION_ATTRIBUTE, XAUTH_USER_NAME, chunk_empty));
    cp->add_attribute(cp, configuration_attribute_create_chunk(
		PLV1_CONFIGURATION_ATTRIBUTE, XAUTH_USER_PASSWORD, chunk_empty));
    *out = cp;
    return NEED_MORE;
}

METHOD(xauth_method_t, process_server, status_t,
    private_xauth_generic_t *this, cp_payload_t *in, cp_payload_t **out)
{
    configuration_attribute_t *attr;
    enumerator_t *enumerator;
    shared_key_t *shared;
    identification_t *id;
    chunk_t user = chunk_empty, pass = chunk_empty;
    status_t status = FAILED;
    int tried = 0;

    enumerator = in->create_attribute_enumerator(in);
    while (enumerator->enumerate(enumerator, &attr))
    {
	switch (attr->get_type(attr))
	{
	    case XAUTH_USER_NAME:
		user = attr->get_chunk(attr);
		break;
	    case XAUTH_USER_PASSWORD:
		pass = attr->get_chunk(attr);
		break;
	    default:
		break;
	}
    }
    enumerator->destroy(enumerator);

    if (!user.ptr || !pass.ptr)
    {
	DBG1(DBG_IKE, "peer did not respond to our XAuth request");
	return FAILED;
    }
    if (user.len)
    {
	id = identification_create_from_data(user);
	if (!id)
	{
	    DBG1(DBG_IKE, "failed to parse provided XAuth username");
	    return FAILED;
	}
	this->peer->destroy(this->peer);
	this->peer = id;
    }
    if (pass.len && pass.ptr[pass.len - 1] == 0)
    {	/* fix null-terminated passwords (Android etc.) */
	pass.len -= 1;
    }

    enumerator = lib->credmgr->create_shared_enumerator(lib->credmgr,
					SHARED_EAP, this->server, this->peer);
    while (enumerator->enumerate(enumerator, &shared, NULL, NULL))
    {
	if (chunk_equals_const(shared->get_key(shared), pass))
	{
	    status = SUCCESS;
	    break;
	}
	tried++;
    }
    enumerator->destroy(enumerator);
    if (status != SUCCESS)
    {
	if (!tried)
	{
	    DBG1(DBG_IKE, "no XAuth secret found for '%Y' - '%Y'",
		 this->server, this->peer);
	}
	else
	{
	    DBG1(DBG_IKE, "none of %d found XAuth secrets for '%Y' - '%Y' "
		 "matched", tried, this->server, this->peer);
	}
    }
    return status;
}

METHOD(xauth_method_t, get_identity, identification_t*,
    private_xauth_generic_t *this)
{
    return this->peer;
}

METHOD(xauth_method_t, destroy, void,
    private_xauth_generic_t *this)
{
    this->server->destroy(this->server);
    this->peer->destroy(this->peer);
    free(this);
}

/*
 * Described in header.
 */
xauth_generic_t *xauth_generic_create_peer(identification_t *server,
					   identification_t *peer,
					   char *profile)
{
    private_xauth_generic_t *this;

    INIT(this,
	.public =  {
	    .xauth_method = {
		.initiate = _initiate_peer,
		.process = _process_peer,
		.get_identity = _get_identity,
		.destroy = _destroy,
	    },
	},
	.server = server->clone(server),
	.peer = peer->clone(peer),
    );

    return &this->public;
}

/*
 * Described in header.
 */
xauth_generic_t *xauth_generic_create_server(identification_t *server,
					     identification_t *peer,
					     char *profile)
{
    private_xauth_generic_t *this;

    INIT(this,
	.public = {
	    .xauth_method = {
		.initiate = _initiate_server,
		.process = _process_server,
		.get_identity = _get_identity,
		.destroy = _destroy,
	    },
	},
	.server = server->clone(server),
	.peer = peer->clone(peer),
    );

    return &this->public;
}

Annex B (addendum) — Multiple subnets via child connections

When you don’t want a full default route but target a split tunnel setup instead, define additional child connections that reuse a base IKE SA:

conn fortigate-base
    keyexchange=ikev1
    authby=xauthpsk
    xauth=client
    xauth_identity=user1
    left=%defaultroute
    leftsourceip=%config
    right=<REDACTED_FGT_IP>
    rightid=@FGT
    ike=aes256-sha256-modp1024!
    esp=aes256-sha256!
    aggressive=yes
    modecfgpull=yes
    dpdaction=restart
    auto=add

# Internal networks
conn fortigate-10-0-0-0_8
    also=fortigate-base
    rightsubnet=10.0.0.0/8
    auto=add

conn fortigate-172-16-0-0_12
    also=fortigate-base
    rightsubnet=172.16.0.0/12
    auto=add

conn fortigate-192-168-100-0_24
    also=fortigate-base
    rightsubnet=192.168.100.0/24
    auto=add

Tip: keep charon { install_routes = yes } in strongswan.conf so the kernel routes are installed automatically for each CHILD_SA.

Compiling latest strongswan 5.9.x on a Debian bullseye works out of the box – but on GNU gcc 15 (kali rolling at the date of this article) is a different challenge – if someone wants full patches contact me at teodor.lupan[at]safebyte.io .

← Back to blog