Reverse-tunneling to bypass a firewall with no fixed IP adress

Set up an SSH reverse tunnel as a `systemd` service, fix proxy DHCP IP address, set-up proxy NAT port forwarding, set up DynHost & DynDNS.

April 04, 2024 • Edited: December 18, 2024

2024   ·   network     ·   tutorial  

The DynHost part requires you to have a domain name from a domain name provider (e.g. OVH).

It is commun to refer to an IP address & port combination with the following syntax: < IP_ADDRESS >:Port, e.g. 127.0.0.0:8888 for localhost and port 8888.

Problem setting

Let’s consider the following scenario:

flowchart LR
    ClientN[
        Client
        Blue
    ]
    FirewallN[[
        <i class="fa-solid fa-shield"></i> Firewall#8201;
    ]]
    WorkstationN["
        Workstation
        Red
    "]
    
    ClientN o-.->|???| WorkstationN
    FirewallN --- WorkstationN
    ClientN x-.-x FirewallN
    


    linkStyle 0 stroke:#ffb3ba

    subgraph "<i class="fa-solid fa-server"></i> Host#8201;#8201;#8201;"
        FirewallN
        WorkstationN
    end
    subgraph "<i class="fa-solid fa-laptop"></i> Client <i class="fa-solid fa-desktop"></i>#8201;#8201;#8201;"
        ClientN
    end

Where we would to connect in ssh from a client called ‘Blue’ to a workstation called ‘Red’ which is part of a corporate or an academic (university) network which is behind a firewall blocking incoming connection an all ports.

Initial search

These are results from an initial google search on the topic. Before getting into the integrated solution.

Bypass firewalls and routers with reverse tunnels

From Bypass firewalls and routers with reverse tunnels by Diogo Silva

(Image from the original post)

Expanding on the OG article:

flowchart TB
	HostAN["
		Host A
		<hr>''Info'' received on port `22` for user `anotherUser`:
		<div style="color:rgb(255,179,186); display:inline-block;">ssh -R 12345:localhost:22</div>
		-> Forward port 12345 to `HostB:port22`
		<hr>Connect to `HostB:22`:
		<div style="color:rgb(186,225,255); display:inline-block;">ssh username@localhost -p 12345</div>
	"]
	RouterAN["
		Router A
		<hr>NAT Port forwarding
		| #8201;Service Port#8201; | #8201;Protocol#8201; | #8201;#8201;#8201;#8201;#8201;IP Address#8201;#8201;#8201;#8201;#8201;#8201; | Internal Port |
		| `6789~6789` | `TCP/IP` | `< Host A IP >` | #8201;#8201;#8201;#8201;#8201;#8201;#8201;#8201;#8201;`22`#8201;#8201;#8201;#8201;#8201;#8201;#8201;#8201;#8201; |
	"]
	RouterBN["
		Router B
	"]
	HostBN["
		Host B (behind firewall)
		<hr><div style="color:rgb(255,179,186); display:inline-block;">ssh -R 12345:localhost:22 anotherUser@routerA -p 6789</div>
	"]


	HostBN -->|<div style="color: LightPink; background-color:grey;opacity:.4;" >ssh -R 12345:localhost:22 anotherUser@routerA -p 6789</div>| RouterBN -->|<div style="color: LightPink; background-color:grey;opacity:.4;" >ssh -R 12345:localhost:22 anotherUser@routerA -p 6789</div>| RouterAN -->|<div style="color: LightPink; background-color:grey;opacity:.4;" >ssh -R 12345:localhost:22 anotherUser@hostA -p 22</div>| HostAN

	linkStyle 0 stroke:#ffb3ba
	linkStyle 1 stroke:#ffb3ba
	linkStyle 2 stroke:#ffb3ba

	HostBN --- RouterBN --- RouterAN --- HostAN

	linkStyle 3 stroke:#bae1ff
	linkStyle 4 stroke:#bae1ff
	linkStyle 5 stroke:#bae1ff

Choosing a port & Solving ‘Warning: remote port forwarding failed for listen port 28’

Solving ‘Warning: remote port forwarding failed for listen port 28’

If you have choosen say port 28 for your tunnel on Host A, you might get something like:

Warning: remote port forwarding failed for listen port 28
From SSH: Troubleshooting “Remote port forwarding failed for listen port” errors – Kamil Maciorowski’s answer

’’

Why does ssh -N -R 2222:localhost:22 <bluehost_user>@<bluehost_ip> result in a “Remote port forwarding failed for listen port” error?

I get this exact warning when I attempt to use a port that is already taken on the remote side.

The output of netstat from bluehost indicates that something is already listening on port 2222 there. It doesn’t show what it is though.

Solutions:

  1. Change 2222 in your ssh invocation to some other port which is not in use on bluehost. Just make it greater than 1023 because regular users can’t bind to well-known ports; otherwise you will get the same warning regardless of whether the port is in use or not.
  2. Or identify the listening process (on bluehost) with sudo lsof -i TCP:2222; terminate or reconfigure it to make the port 2222 available.

’’

Referred netstat cmd on bluehost:

netstat --on --wide | grep 22

Expanding on the OG answer:

Choosing a port

Check the Wikipedia – tcp port list

Example of potentially available ports: 9104~9118

(Image from Wikipedia)

Avoid to keep an active shell subsequent to the back-tunnel launch

We have seen how to launch a reverse tunnel from HostB, however, with the current cmd, if the shell is close the cmd will halt and the tunnel will close.

We get some first hints:

From SSH: tunnel without shell on ssh server – Adrien M.’s answer

’’ As said in other posts, if you don’t want a prompt on the remote host, you must use the -N option of SSH. But this just keeps SSH running without having a prompt, and the shell busy.

You just need to put the SSH’ing as a background task with the & sign :

ssh -N -L 8080:ww.xx.yy.zz:80 user@server &

This will launch the ssh tunnelling in the background. But some messages may appear, especially when you try to connect to a non-listening port (if you server apache is not launched). To avoid these messages to spawn in your shell while doing other stuff, you may redirect STDOUT/STDERR to the big void :

ssh -N -L 8080:ww.xx.yy.zz:80 user@server >/dev/null 2>&1 &

’’

In a nutshell:

To not get a shell prompt on Remote-forwarding:

ssh -N -L 8080:ww.xx.yy.zz:80 user@server

To not get the busy shell, put it in the background:

ssh -N -L 8080:ww.xx.yy.zz:80 user@server &

But actually, the better solution is delivered:

From SSH port forwarding without session – ghoti’s answer

’’ Rather than backgrounding an active TTY, you can just let SSH take care of things by itself.
My suggestion is this:

ssh -fNT -R $rport:dc-bb7925e7:$camport -p 25 server@192.168.178.20

The first three options are:

-f    Requests ssh to go to background just before command execution.
-N    Do not execute a remote command.
-T    Disable pseudo-tty allocation.

The -f option is the one that actually backgrounds things, but -N and -T will save resources which you don’t need to allocate for an SSH session whose sole purpose is to carry your tunnel.

Also note that some of these options can be added to a custom profile in your ~/.ssh/config file, in case you feel it would be preferable to put more of your static configuration into static configuration files rather than scripts. The RemoteForward config file option is equivalent to the -R command line option.

See also:

’’

Controlling a detached ssh connection with ControlMaster

From How do I kill a backgrounded/detached ssh session? – ghoti’s answer

Initial cmd trigering the question:

ssh -f -N -L localhost:12345:otherHost:12345 otherUser@OtherHost
synergyc localhost

’’

With all due respect to the users of pgrep, pkill, ps | awk, etc, there is a much better way.

Consider that if you rely on ps -aux | grep ... to find a process you run the risk of a collision. You may have a use case where that is unlikely, but as a general rule, it’s not the way to go.

SSH provides a mechanism for managing and controlling background processes. But like so many SSH things, it’s an “advanced” feature, and many people (it seems, from the other answers here) are unaware of its existence.

In my own use case, I have a workstation at home on which I want to leave a tunnel that connects to an HTTP proxy on the internal network at my office, and another one that gives me quick access to management interfaces on co-located servers. This is how you might create the basic tunnels, initiated from home:

$ ssh -fNT -L8888:proxyhost:8888 -R22222:localhost:22 officefirewall
$ ssh -fNT -L4431:www1:443 -L4432:www2:443 colocatedserver

These cause ssh to background itself, leaving the tunnels open. But if the tunnel goes away, I’m stuck, and if I want to find it, I have to parse my process list and home I’ve got the “right” ssh (in case I’ve accidentally launched multiple ones that look similar).

Instead, if I want to manage multiple connections, I use SSH’s ControlMaster config option, along with the -O command-line option for control. For example, with the following in my ~/.ssh/config file,

host officefirewall colocatedserver
    ControlMaster auto
    ControlPath ~/.ssh/cm_sockets/%r@%h:%p

the ssh commands above, when run, will leave spoor in ~/.ssh/cm_sockets/ which can then provide access for control, for example:

$ ssh -O check officefirewall
Master running (pid=23980)
$ ssh -O exit officefirewall
Exit request sent.
$ ssh -O check officefirewall
Control socket connect(/home/ghoti/.ssh/cm_socket/ghoti@192.0.2.5:22): No such file or directory

And at this point, the tunnel (and controlling SSH session) is gone, without the need to use a hammer (kill, killall, pkill, etc).

Bringing this back to your use-case…

You’re establishing the tunnel through which you want syngergyc to talk to syngergys on TCP port 12345. For that, I’d do something like the following.

Add an entry to your ~/.ssh/config file:

Host otherHosttunnel
    HostName otherHost
    User otherUser
    LocalForward 12345 otherHost:12345
    RequestTTY no
    ExitOnForwardFailure yes
    ControlMaster auto
    ControlPath ~/.ssh/cm_sockets/%r@%h:%p

Note that the command line -L option is handled with the LocalForward keyword, and the Control{Master,Path} lines are included to make sure you have control after the tunnel is established.

Then, you might modify your bash script to something like this:

#!/bin/bash

if ! ssh -f -N otherHosttunnel; then
    echo "ERROR: couldn't start tunnel." >&2
    exit 1
else
    synergyc localhost
    ssh -O exit otherHosttunnel
fi

The -f option backgrounds the tunnel, leaving a socket on your ControlPath to close the tunnel later. If the ssh fails (which it might due to a network error or ExitOnForwardFailure), there’s no need to exit the tunnel, but if it did not fail (else), synergyc is launched and then the tunnel is closed after it exits.

You might also want to look in to whether the SSH option LocalCommand could be used to launch synergyc from right within your ssh config file.

’’

Solution

Thus, the solution would simply be to open a reverse tunnel from Workstation to Proxy, and then connect from Client to Proxy on the right port to actually access Workstation:

flowchart LR
    ClientN[
        Client
        Blue
    ]
    FirewallN[
        <i class="fa-solid fa-shield"></i> Firewall#8201;
    ]
    WorkstationN["
        Workstation
        Red
    "]
    ProxyN["
        Proxy
        Pi
    "]
    
    ClientN o-.->|???| WorkstationN
    FirewallN --- WorkstationN
    ClientN x-.-x FirewallN
    ClientN --> ProxyN 
    WorkstationN --> ProxyN


    linkStyle 0 stroke:#ffb3ba



    subgraph "<i class="fa-solid fa-server"></i> Host#8201;#8201;#8201;"
        FirewallN
        WorkstationN
    end

    subgraph "<i class="fa-solid fa-laptop"></i> Client <i class="fa-solid fa-desktop"></i>#8201;#8201;#8201;"
        ClientN
    end

    subgraph "<i class="fa-solid fa-server"></i> Proxy#8201;#8201;#8201;"
        ProxyN
    end

There are then 2 aspects to deal with:

  • On Workstation, stablishing a robust and always on reverse tunnel
  • Configuring a Proxy at home to which Workstation could connect: the ‘‘at home’’ here, actually implies an hidden yet present constraint, that is the fact that, at home, your external IP adress, the one of your router, might actually change now and then.

Let’s first configure the Proxy side of things, before we dive into the Workstation side where two approaches will be presented:

  • The babarian one: manual with bash scripts and cron
  • The nice one using systemd

Proxy side configuration

flowchart LR
    ClientN[
        <i class="fa-solid fa-laptop"></i>Client#8201;#8201;#8201;
        Blue
    ]
	ProxyN["
		Proxy
		Pi
		<hr>< PROXY_IP >
	"]
	RouterN["
        <i class="fa-solid fa-house-signal"></i><i class="fa-solid fa-shield"></i> Router#8201;#8201;#8201;
        <hr>< ROUTER_IP >
	"]
    WorkstationN["
        <i class="fa-solid fa-server"></i>Workstation#8201;#8201;#8201;
        Red
    "]
	DHCPN["
		DHCP
		<hr>Assign < PROXY_IP >
	"]
	NATN["
		NAT
		<hr>Direct incoming Port request to < Internal IP >:Port
	"]
	DynHostN["
		DynHost
		<hr>< SUBDOMAIN_URL > to < ROUTER_IP >
	"]
	ddclientN["
		ddclient
	"]
    UFWn["
		<i class="fa-solid fa-shield"></i>UFW#8201;#8201;#8201;
	"]
    
    subgraph "<i class="fa-solid fa-server"></i>Proxy#8201;#8201;#8201;"
		UFWn
		ProxyN
		subgraph "<i class="fa-solid fa-house-signal"></i><i class="fa-solid fa-shield"></i>Router#8201;#8201;#8201;#8201;#8201;#8201;"
			RouterN
			subgraph "Config"
				DHCPN
				NATN
			end
			
		end
		ddclientN
	end

	subgraph "OVH"
		subgraph "DNSZone"
			DynHostN
		end
	end


	RouterN --- Config
	NATN --> UFWn
	DHCPN --> UFWn
	UFWn --- ProxyN
	DynHostN -->|< ROUTER_IP >:Port| RouterN
	ProxyN --- ddclientN --> DynHostN 
	ClientN --> DNSZone
	WorkstationN --> DNSZone

Behind the router usually actually lies a subnetwork. That means that the IP address assigned to devices on the router network are actually IP addresses of an internal network, and are not readily accessible from the outside.

Proxy device choice

First a word on the choice of a raspberry pi for the Proxy.
I consider that the workstation is always on, otherwise we would need another accessible device from the outside to Wake-On-Lan the Workstation.
Now, regarding the Proxy:
Either you want to enable the tunnel at use and Wake-On-Lan the Proxy, or have a Proxy which can be on 24/7.
I chose the latest, hence a low-power consumption device meant to be always on.

Set-up the raspberry-pi

Following the raspberry pi installer guide, install the rpi imager:

$ sudo apt install rpi-imager

Then launch it and follow the instructions. Choose:

  • default debian
  • User: <ur_username>
  • Passwd: <passwd>
  • Enable SSH and select port 22

If: ERROR: Couldn’t determine iptables version From How to fix ERROR: Couldn’t determine iptables version:

update-alternatives --list iptables

If result is:

/usr/sbin/iptables-legacy
/usr/sbin/iptables-nft

Then:

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy

If: Can’t initialize iptables table `filter’: Table does not exist (do you need to insmod?) From Can’t initialize iptables table `filter’: Table does not exist (do you need to insmod?):
Then:

sudo apt-get install rpi-update
sudo chmod +x /sbin/depmod
sudo sudo rpi-update

In case, you need to enable other ports on the raspberry pi, following:

Install ufw and use ufw to allow them (e.g 26) there:

sudo apt-get install ufw
sudo ufw enable
sudo ufw allow 26

Set-up the Proxy router

'’If you are testing from inside the network it is likely that your router does not support NAT loopback.’’ Port Forwarding to enable ssh remote access is not working

We have to tell the router to:

  • keep the internal IP address of the Proxy fixed
  • tell it to redirect request on port 22 to < PROXY_IP > Port 22

We also need to tell ‘the internet’ what is the address of the router, because there is no guarantee that the router IP address does not change if your Internet Service Provider (ISP) did not give you a fixed IP address.

Do not forget to confirm the change (for the router I am using, that confirmation button requires a total scrool down to find), otherwise the change is not applied.

Dynamic Host Configuration Protocol (DHCP)

The DHCP is a dynamic allocation of ‘internal’ IPs to devices connected to the router.

To check the currently assigned IPs in the DHCP, on the router, check the following:
DHCP assigment list

No. IP Address Host Information MAC Address Assigned Status
X 192.168.219.XXX Hostname XX:XX:XX:XX:XX:XX Automatic allocation/IP fixed allocation
3 192.168.219.2 pi AA:AA:AA:AA:AA:AA Automatic allocation

As you can notice, the status can be either Automatic allocation or IP fixed allocation.

By default, the status for new connected devices is Automatic allocation. This means that the router will assign an address chosen in the available pool of address (in my case, I believe the Subnet mask is 255.255.255.0 meaning that all the address of the subnetwork will be of the from 192.168.219.XXX).This also means that there is no actual guarantee that this IP (here, 192.168.219.2) will not change.

Thus, we have to set-up a DHCP fixed allocation for the rapsberry pi so that the router Port forwarding, in the NAT, does not fail.

To fix the IP for the rapsberry pi, go to DHCP static assigment list, and add a new one:
DHCP static assigment list

No. On/Off Hardware address IP address
X [x] ON [] OFF XX:XX:XX:XX:XX:XX 192.168.219.XXX
3 [x] ON [] OFF AA:AA:AA:AA:AA:AA 192.168.219.2

There is usually a search option, so that you can directly select an already connected device, instead of only to copy and paste the MAC adress from the previous page.

Network Address Translation (NAT)

To forward a port to the Proxy device, add the last line to your configuration, on the router, in: NAT Port forwarding

Service Port Protocol IP Address Internal Port
XXXX~XXXX TCP/IP XXX:XXX:XXX:XXX XXXX
22~22 TCP/IP 192.168.219.2 22

Here:

  • Service Port: is the range of port for whose request are to be redirected
  • IP Address: is the subnet IP address of the device to which to redirect the request, so < PROXY_IP >
  • Internal Port: is the port on the device to which the request is to be redirected, 22

Basically, to access the pi from the outside, one need to enter the IP address of the router A with the correct port, any such request, provided the port forwarding is correctly configured, will be passed to the local fixed IP on the designated port. Here, any request made on < ROUTER_IP >:22 will be transfered to 192.168.219.2:22 which is the local/internal IP address of the pi and port 22.

Domain Name System (DNS), DynDNS/DynHost

A DNS allows to link subdomains to IP addresses, hence you type a ‘nice’ Uniform Resource Locator (URL) https://www.google.com instead of an actual IP address, and because DNS Zones exist with the correct configuration, under the hood, you are redirected to the correct IP address.

Because the IP address of the router might also dynamic, we are not guaranteed that it will stay the same. To avoid that problem, we can set up a Dynamic DNS (DynDNS). That is we are gonna link the router IP to a subdomain and update the IP address if it changes.

To do so, we need:

  • make the Proxy device send the < ROUTER_IP > to domain provider of our choice, for me, OVH (using DynHost, DynDNS)
  • associate a subdomain of our choice to the address sent by the Proxy device (using ddclient)
Set up a DynDNS on OVH

(Source: Set up a DynDNS on OVH)

Keep in mind or on note, the following configurations for the subsequent set-ups:

  • username
  • password
  • subdomain

As these will be needed for the Dyn. Host Username set-up, the Dyn. DNS set-up, AND the ddclient.

Let’s create a Dyn. DNS.

To do so, we need to define a DynHost Username, go to (in Web Cloud on OVH) DNS zones/< UR_DOMAIN >/DynHost, Manage accessand Create a username like so:

ID Zone Sub-domain
< UR_DOMAIN >-< SUBDOMAIN > < UR_DOMAIN > < SUBDOMAIN >
blissfox.xyz-proxy blissfox.xyz proxy

Then, we need to define DynHost record, still in (in Web Cloud on OVH) DNS zones/< UR_DOMAIN >/DynHost, Add a DynHost record like so:

DynHost Target
proxy.blissfox.xyz 0.0.0.0

The username subdomain should match the subdomain given in the definition of the Dyn. DNS.

The IP address to give at first is 0.0.0.0. It will change automatically once set up the Proxy correctly.

With this we have, a DynDNS in place. Now, we need to put in-place the mechanism to update the IP address linked with the subdomain.

ddclient on the Proxy

To update the IP address linked with the subdomain, I used these references:

And made the following tutorial.

First, install ddclient:

apt install ddclient

When installing ddclient, we are directly asked for the config. of the target DNS and the IP you want to push, in order:

  • service=other
  • protocol=dyndns2
  • use=web
  • server=www.ovh.com
  • login=[usernameDynHost]
  • password=[passwordDynHost]
  • [sudbdomain] (e.g with the example above proxy.blissfox.xyz)

Regarding use, you are asked whether you want to use web or nat. Actually:

  • web: will get the global IP address used, the same a web loop-up-ip would. It is in fact, the IP address of the router. So, the IP address, you need if you are sitting outside of the network of the router.
  • nat: will get the local/internal IP address. It is the one you would get by doing ifconfig in a terminal, the one used in the local/internal network of the router. Thus, anything outside of this network or of the router, would have no idea what these are.

We want the web option, because we want the global IP address to access it from outside of the network, and anyway, inside of the network, we can directly use the internal/local IP address as we have fixed it in the DHCP.
If you messed up or just want to check the configuration afterwards, you can:

sudo vim /etc/ddclient.conf

/etc/ddclient.conf

# Configuration file for ddclient generated by debconf
#
# /etc/ddclient.conf

protocol=dyndns2 \
use=web, web=https://api.ipify.org/ \
server=www.ovh.com \
login=[usernameDynHost] \
password=[passwordDynHost] \
[sudbdomain]

After modification of /etc/ddclient.conf:

sudo ddclient
sudo service ddclient restart
Back-checking the DynHost

If you have correctly configured the DynHost and ddclient, you should now see that in (in Web Cloud on OVH) DNS zones/< UR_DOMAIN >/DynHost, you now have

DynHost Target
proxy.blissfox.xyz < ROUTER_IP >

Workstation side configuration

As stated earlier, two approaches will be presented:

  • The babarian one: using ssh, manual bash scripts and crontab
  • The better one: using autossh as a systemd service

In any case, you should create some ssh keys for the connection automation, otherwise you will be prompted for a password.

Set up SSH keys

Following:

On workstation:

ssh-keygen

Choose:

  • A name: ‘workstation_to_pi’
  • A location: Let it be in ~/.ssh/
  • A password/passphrase: choose empty, that is just press Enter when asked for a password

Two keys have now be generated: workstation_to_pi & workstation_to_pi.pub

To copy the key to the Proxy:

ssh-copy-id -i ~/.ssh/workstation_to_pi proxy_user@proxy.blissfox.xyz

The key will be copied/registered in ~/.ssh/authorized_keys.

Barbarian way: ssh, manual bash scripts & crontab

  • Making scripts to:
    • Check if the back-tunnel is already launched if not launch the back-tunnel
    • Kill any old back-tunnel (killing only one tunnel if existence of at least one is found) and relaunch the back-tunnel
  • Setting up the crontab to launch the two scripts:
    • Launching if not existant every minute
    • Killing the old back-tunnel if exists and relaunching a new tunnel every 30 minutes (originally set-up for every hour but it seems to be too much time, and the connection stops working before that, so reduced to 30 min. for convenience sake)

Launching a reverse-tunnel

My config In my config, to launch a back-tunnel connection from the server to the client:

ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22

Back tunnel is created on port 9108 of the raspberry pi.

If you need to precise the ssh key, e.g. because:

OpenSSH only allows a maximum of five keys to be tried authomatically. If you have more keys, you must specify which key to use using the -i option to ssh. – SSH Copy ID for Copying SSH Keys to Servers – Troubleshooting

You can use the -i option of the command (-i <path_to_ur_private_key>):

ssh -N -ft -R 9108:localhost:24 -i ~/.ssh/workstation_to_pi proxy_user@proxy.blissfox.xyz -p 22

Making scripts

Get the first PID linked to the back-tunnel

The following script fetches the very first PID associated with the back-tunnel: check_sshR_pid.sh

#!/bin/bash

# https://tldp.org/LDP/abs/html/string-manipulation.html

my_tmp=$(ps -aux | grep -v grep | grep 'ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22')
if [ -z "$my_tmp" ]
then
	echo 'Process not running, no pid to return.'
else
	#echo "$my_tmp"
	my_idx=$(expr index "$my_tmp" ' ')
	#echo "$my_idx"
	my_tmp=${my_tmp:my_idx+1}
	#echo "$my_tmp"
	my_idx=$(expr index "$my_tmp" ' ')
	echo "$my_idx"
	while [ "$my_idx" -eq 1 ]
	do
		echo 'removing another space'
		my_tmp=${my_tmp:my_idx}
		my_idx=$(expr index "$my_tmp" ' ')
	done
	echo "$my_tmp"
	my_idx=$(expr index "$my_tmp" ' ')
	echo "$my_idx"
	echo "PID of ssh back tunneling:${my_tmp:0:my_idx}"
fi

It works because I checked that the effective shape of the cmd in:

ps -aux | grep -i ssh

is indeed:

ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22
Check if the back-tunnel is already launched if not launch the back-tunnel

If for whatever reason the back-tunnel is halted of the workstation, it should restart by itself. Or if we reboot the workstation, the back-tunnel should also restart by itself otherwise the ssh connection is lost.
The script in order to do so is the following:
cron_sshR.sh

workstation_user@workstation:~$ cat  cron_sshR.sh
#!/bin/bash

my_tmp=$(ps -aux | grep -v grep | grep 'ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22')
if [ -z "$my_tmp" ]
then
	ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22
fi

Now to make its execution automatic, it should be provided to the crontab (see the crontab section for this).

Kill any old back-tunnel (killing only one tunnel if existence of at least one is found) and relaunch the back-tunnel

The back-tunnel needs to be restarted at regular intervals otherwise the connection is lost. To do so, we first need (considering there should always be at most one such back-tunnel) to kill the previous one if it exists, and relaunch a new one.
The script in order to do so is the following:
cron_kill_relaunch_sshR.sh

#!/bin/bash

# https://tldp.org/LDP/abs/html/string-manipulation.html

my_tmp=$(ps -aux | grep -v grep | grep 'ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22')
if [ -z "$my_tmp" ]
then
	ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22
else
	my_idx=$(expr index "$my_tmp" ' ')
	my_tmp=${my_tmp:my_idx+1}
	my_idx=$(expr index "$my_tmp" ' ')
	while [ "$my_idx" -eq 1 ]
	do
		my_tmp=${my_tmp:my_idx}
		my_idx=$(expr index "$my_tmp" ' ')
	done
	my_idx=$(expr index "$my_tmp" ' ')
	kill ${my_tmp:0:my_idx}
	ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22
fi

Now to make its execution automatic, it should be provided to the crontab (see the crontab section below).

crontab

For info on the crontab, one can refer to:

To launch scripts at regular intervals, we have to set-up crontab. But first, we must maake our scripts executable by adding -x rights to the scripts:

chmod +x cron_sshR.sh
chmod +x cron_kill_relaunch_sshR.sh

To check the current crontab:

crontab -l

To edit the crontab:

crontab -e

To launch cron_sshR.sh every minute and cron_kill_relaunch_sshR.sh every 30 minutes, edit as follows:

# m h  dom mon dow   command

*/1* * * * * /home/workstation_user/cron_sshR.sh
*/30 * * * * /home/workstation_user/cron_kill_relaunch_sshR.sh

To check if the schedule is correct, one can use this site.

Better way: autossh as a systemd service

  • Use autossh: autossh(1) - Linux man page: ‘‘autossh is a program to start a copy of ssh and monitor it, restarting it as necessary should it die or stop passing traffic.’’
  • Configure autossh as a systemd service

Following:

Use autossh

Let’s first open an SSH port (Port 24) on workstation in /etc/ssh/sshd_config:

...
Include /etc/ssh/sshd_config.d/*.conf

Port 24
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
...

After changing the ssh config file, restart the ssh service using either service or systemctl:

sudo service ssh restart

You might also have to allow the port for ufw.

sudo ufw allow 24

We can then use:

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -fN -T -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22

To open a reverse tunnel on Proxy port 9108. Where from autossh docs (autossh(1) - Linux man page, OpenBSD manual page server – ssh_config#ServerAliveInterval):

  • -M : port[:echo_port]

    specifies the base monitoring port to use. Without the echo port, this port and the port immediately above it ( port + 1) should be something nothing else is using. autossh will send test data on the base monitoring port, and receive it back on the port above. For example, if you specify “-M 20000”, autossh will set up forwards so that it can send data on port 20000 and receive it back on 20001.

    Alternatively, a port for a remote echo service may be specified. This should be port 7 if you wish to use the standard inetd echo service. When an echo port is specified, only the specified monitor port is used, and it carries the monitor message in both directions.

    Many people disable the echo service, or even disable inetd, so check that this service is available on the remote machine. Some operating systems allow one to specify that the service only listen on the localhost (loopback interface), which would suffice for this use.

    The echo service may also be something more complicated: perhaps a daemon that monitors a group of ssh tunnels.

    Setting the monitor port to 0 turns the monitoring function off, and autossh will only restart ssh upon ssh’s exit. For example, if you are using a recent version of OpenSSH, you may wish to explore using the ServerAliveInterval and ServerAliveCountMax options to have the SSH client exit if it finds itself no longer connected to the server. In many ways this may be a better solution than the monitoring port.

  • -f :causes autossh to drop to the background before running ssh. The -f flag is stripped from arguments passed to ssh. Note that there is a crucial a difference between -f with autossh, and -f with ssh: when used with autossh ssh will be unable to ask for passwords or passphrases. When -f is used, the “starting gate” time (see AUTOSSH_GATETIME) is set to 0.

  • ServerAliveCountMax

    Sets the number of server alive messages (see below) which may be sent without ssh(1) receiving any messages back from the server. If this threshold is reached while server alive messages are being sent, ssh will disconnect from the server, terminating the session. It is important to note that the use of server alive messages is very different from TCPKeepAlive (below). The server alive messages are sent through the encrypted channel and therefore will not be spoofable. The TCP keepalive option enabled by TCPKeepAlive is spoofable. The server alive mechanism is valuable when the client or server depend on knowing when a connection has become unresponsive.

    The default value is 3. If, for example, ServerAliveInterval (see below) is set to 15 and ServerAliveCountMax is left at the default, if the server becomes unresponsive, ssh will disconnect after approximately 45 seconds.

  • ServerAliveInterval

    Sets a timeout interval in seconds after which if no data has been received from the server, ssh(1) will send a message through the encrypted channel to request a response from the server. The default is 0, indicating that these messages will not be sent to the server. And from ssh docs:

  • -N : Do not execute a remote command. This is useful for just forwarding ports (protocol version 2 only).

  • -L : [bind_address:]port:host:hostport

    Specifies that the given port on the local (client) host is to be forwarded to the given host and port on the remote side. This works by allocating a socket to listen to port on the local side, optionally bound to the specified bind_address. Whenever a connection is made to this port, the connection is forwarded over the secure channel, and a connection is made to host port hostport from the remote machine. Port forwardings can also be specified in the configuration file. IPv6 addresses can be specified with an alternative syntax:
    [bind_address/]port/host/hostport or by enclosing the address in square brackets. Only the superuser can forward privileged ports. By default, the local port is bound in accordance with the GatewayPorts setting. However, an explicit bind_address may be used to bind the connection to a specific address. The bind_address of ‘‘localhost’’ indicates that the listening port be bound for local use only, while an empty address or ‘*’ indicates that the port should be available from all interfaces.

  • -f : Requests ssh to go to background just before command execution. This is useful if ssh is going to ask for passwords or passphrases, but the user wants it in the background. This implies - -n. The recommended way to start X11 programs at a remote site is with something like ssh -f host xterm.

    If the ExitOnForwardFailure configuration option is set to ‘‘yes’’, then a client started with -f will wait for all remote port forwards to be successfully established before placing itself in the background.

  • -t : Force pseudo-tty allocation. This can be used to execute arbitrary screen-based programs on a remote machine, which can be very useful, e.g. when implementing menu services. Multiple -t options force tty allocation, even if ssh has no local tty.
  • -T : Disable pseudo-tty allocation.

  • -R : [bind_address:]port:host:hostport Specifies that the given port on the remote (server) host is to be forwarded to the given host and port on the local side. This works by allocating a socket to listen to port on the remote side, and whenever a connection is made to this port, the connection is forwarded over the secure channel, and a connection is made to host port hostport from the local machine.

    Port forwardings can also be specified in the configuration file. Privileged ports can be forwarded only when logging in as root on the remote machine. IPv6 addresses can be specified by enclosing the address in square braces or using an alternative syntax:
    [bind_address/]host/port/hostport.

    By default, the listening socket on the server will be bound to the loopback interface only. This may be overridden by specifying a bind_address. An empty bind_address, or the address ‘*’, indicates that the remote socket should listen on all interfaces. Specifying a remote bind_address will only succeed if the server’s GatewayPorts option is enabled (see sshd_config(5)).

    If the port argument is ‘0’, the listen port will be dynamically allocated on the server and reported to the client at run time.

  • -M : Places the ssh client into ‘‘master’’ mode for connection sharing. Multiple -M options places ssh into ‘‘master’’ mode with confirmation required before slave connections are accepted. Refer to the description of ControlMaster in ssh_config(5) for details.

If you need to precise the ssh key, e.g. because:

OpenSSH only allows a maximum of five keys to be tried authomatically. If you have more keys, you must specify which key to use using the -i option to ssh. – SSH Copy ID for Copying SSH Keys to Servers – Troubleshooting

You can use the -i option of the command (-i <path_to_ur_private_key>):

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -fN -T -R 9108:localhost:24 -i ~/.ssh/workstation_to_pi proxy_user@proxy.blissfox.xyz -p 22

Configure autossh as a systemd service

Before making it into a service, please first test launching the ssh cmd with the arguments, then test the autossh cmd with the arguments, to check that it works, and that once on Proxy you can access Workstation launching:

ssh workstation_user@localhost -p 9108

cf. Launching a reverse-tunnel

ssh -N -ft -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22

or if you need to precise the key:

ssh -N -ft -R 9108:localhost:24 -i ~/.ssh/workstation_to_pi proxy_user@proxy.blissfox.xyz -p 22

cf. Use autossh

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -fN -T -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22

or if you need to precise the key:

autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -fN -T -R 9108:localhost:24 -i ~/.ssh/workstation_to_pi proxy_user@proxy.blissfox.xyz -p 22

First, create the file autossh-tunnel.service below as a normal user

[Unit]
Description=AutoSSH tunnel service Remote port 9108 to local 24
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=0

[Service]
User=workstation_user
Environment="AUTOSSH_GATETIME=0"
ExecStart=/usr/bin/autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -o "ConnectTimeout 10" -o "ExitOnForwardFailure yes" -N -q -T -R 9108:localhost:24 proxy_user@proxy.blissfox.xyz -p 22
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

If you need to precise the ssh key, e.g. because:

OpenSSH only allows a maximum of five keys to be tried authomatically. If you have more keys, you must specify which key to use using the -i option to ssh. – SSH Copy ID for Copying SSH Keys to Servers – Troubleshooting

You can use the -i option of the command (-i <path_to_ur_private_key>), change the corresponding line to:

ExecStart=/usr/bin/autossh -M 0 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 3" -o "ConnectTimeout 10" -o "ExitOnForwardFailure yes" -N -q -T -R 9108:localhost:24 -i /home/workstation_user/.ssh/workstation_to_pi proxy_user@proxy.blissfox.xyz -p 22

Change mod to 777:

chmod 777 autossh-tunnel.service

Then move it to /etc/systemd/system/:

$ sudo mv autossh-tunnel.service /etc/systemd/system/

So that, now the file is /etc/systemd/system/autossh-tunnel.service.

Lastly enable it:

$ sudo systemctl enable autossh-tunnel.service

And reload the sevices:

systemctl daemon-reload
systemctl start autossh-tunnel.service
systemctl enable autossh-tunnel.service

Using Proxy (port 9108) in practice

You will have to create and dispatch ssh keys for the client you’re using to connect to the proxy and to the workstation.

Initial Search

  • From Correct ssh config file settings to tunnel to a 3rd machine

    ’‘ProxyJump ~/.ssh/config example

    ~/.ssh/config

    Host server1
  Hostname server1.example.com
  IdentityFile ~/.ssh/id_rsa

Host server2_behind_server1
  Hostname server2.example.com
  IdentityFile ~/.ssh/id_rsa
  ProxyJump server1

    Connect with

    ssh server2_behind_server1 -v

    Add -v for verbose output

    ProxyJump -J Command line example

    ~/.ssh/config

    Host server1
  Hostname server1.example.com
  IdentityFile ~/.ssh/id_rsa

Host server2
  Hostname server2.example.com
  IdentityFile ~/.ssh/id_rsa

    Connect with

    ssh server2 -J server1 -v

    Or use -o

    ssh server2 -o 'ProxyJump server1' -v

    ’’

  • VS code SSH tunnel

On Ubuntu

~/.ssh/config:

# Read more about SSH config files: https://linux.die.net/man/5/ssh_config
Host proxy                                                                                                                                       
		Hostname proxy.blissfox.xyz
        User proxy_user
        Port 22

Host workstation_sshR
        Hostname localhost
        User workstation_user
        Port 9108
        ProxyJump proxy

On Windows

MobaXterm reverse-tunneling SSH config

Following:

The one-liner to ssh via proxy with the old option ProxyCommand instead of ProxyJump:

ssh -o ProxyCommand="ssh -W %h:%p proxy" workstation_user@localhost -p 9108

The equivalent configuration for the .ssh/config:

config file
Host proxy
	HostName proxy.blissfox.xyz
	User proxy_user
	Port 22

Host workstation_sshR
	HostName localhost
	User workstation_user
	Port 9108
	ProxyCommand ssh proxy -W %h:%p

Using WSL Ubuntu

~/.ssh/config

# Read more about SSH config files: https://linux.die.net/man/5/ssh_config

# Host alias
#     HostName hostname
#     User user

Host proxy
        HostName proxy.blissfox.xyz
        User proxy_user
        Port 22
        IdentityFile /home/wsl_ubuntu_user/.ssh/wsl_host_to_proxy

Host workstation_sshR
        HostName localhost
        User workstation_user
        Port 9108
        IdentityFile /home/wsl_ubuntu_user/.ssh/wsl_host_to_workstation
        ProxyJump proxy

Firefox Proxy

Following:

The cmd to launch the ssh on local port (be it: Windows-MobaXterm tunneling but on the cmdline; or Ubuntu), provided you have configured your ~/.ssh/config correctly:

ssh -N -D 3218 workstation_sshR

-D [bind_address:]port
	Specifies a local “dynamic” application-level port
	forwarding.  This works by allocating a socket to listen
	to port on the local side, optionally bound to the
	specified bind_address.  Whenever a connection is made to
	this port, the connection is forwarded over the secure
	channel, and the application protocol is then used to
	determine where to connect to from the remote machine.
	Currently the SOCKS4 and SOCKS5 protocols are supported,
	and will act as a SOCKS server.  Only root can forward
	privileged ports.  Dynamic port forwardings can also be
	specified in the configuration file.

	IPv6 addresses can be specified by enclosing the address
	in square brackets.  Only the superuser can forward
	privileged ports.  By default, the local port is bound in
	accordance with the GatewayPorts setting.  However, an
	explicit bind_address may be used to bind the connection
	to a specific address.  The bind_address of “localhost”
	indicates that the listening port be bound for local use
	only, while an empty address or ‘*’ indicates that the
	port should be available from all interfaces.
-N      Do not execute a remote command.  This is useful for just
	forwarding ports.  Refer to the description of
	SessionType in ssh_config(5) for details.

Now, ssh to workstation_sshR is on local port 3218 of your client.

Thus, all that is left is to use that port as a proxy in Firefox. In Settings>General>Network Settings>Settings...:

Once you validate with Ok, you will now be browsing in Firefox as if you were on Workstation.

Miscellaneous

Time a connection liveliness time

This part is only meant to help testing whether a connection stays alive or not and more importantly for how long.

Following: Find The Execution Time Of A Command Or Process In Linux

One can simply use the time cmd:

time ssh blissfox_lab_sshR

Better launch it in a screen, tmux or byobu detachable session though.

Basically, if the connection gets cut, ssh blissfox_lab_sshR will exit, and time will give the amount of time that the cmd has run for.

Check out that the service is listening on the port

How to check if port is in use on Linux or Unix:

sudo lsof -i -P -n | grep LISTEN

Generate ssh key in Windows

In Windows shell cmd run as an admin (after checking in app-> optional features-> openssh):

ssh-keygen

You will then have to put/move/copy the keys in the right places.

MobaXtem ssh config location

MobaXTerm - SSH Key authentication – Wyck’s answer:

’’ If you already have keys setup via Putty or something else the easiest thing to do is to copy and paste them into the MobaXterm’s ssh directory.

So copy them then paste the keys into: C:\Users\%USERNAME%\Documents\MobaXterm\home\.ssh

Note:

Putty’s default location is usually C:\Users\.ssh

Cygwin’s default location is usually C:\cygwin64\home\%USERNAME%\.ssh

’’



Please do not hesitate to leave a comment below or dm me if you have any feedback, whether it is about a mistake I made in the articles or suggestions for improvements.

    Other articles:

  • Self-hosted ClearML Server behind a firewall
  • Docker, Docker Compose & Nvidia GPUs
  • dummy