Update to redwood idle session management following August 20, 2024 downtime

The team at the Center for High Performance Computing (CHPC) is aware that recent changes to the redwood cluster have disrupted workflows and caused unexpected challenges for some users. During the redwood cluster downtime on August 20, the CHPC implemented systems to lock idle connections to improve the security posture of the Protected Environment. For SSH connections, we implemented a "transparent Screen" session that automatically locks after a period of inactivity. This change, however, has had unexpected impacts on certain applications and on terminal behavior.

Following the Standards for the Protection of Electronic Protected Health Information, we implemented the screen lock as a procedure to manage idle sessions. Based on feedback since the downtime, our implementation proved to have significant challenges for some users due to how Screen affects the session. In response, we have prepared an alternate approach that terminates idle sessions, but does not interfere with users’ terminal behavior. This, apart from the enforcement of idle session termination, would be similar to what users experienced prior to the downtime/upgrade. Our new implementation utilizes systemd-logind to terminate idle sessions—those with no direct user activity for 15 minutes—based on the StopIdleSessionSec parameter.

We understand that users may need to run programs that persist beyond the 15-minute limit on idle sessions (consistent with CHPC login node usage policy). Users may utilize the Linux tools Screen or tmux for applications that need to persist through idle session termination. Please note that to have a Screen or tmux session persist the idle session termination, they need to be started up a bit differently:

Example

Start a Screen session:

systemd-run --scope --user screen

Start a tmux session:

systemd-run --scope --user tmux

We have set up 2 system aliases to help users to have quick access to these: pe-screen and pe-tmux will be defined on fresh logins. Users are encouraged to make their own according to the parameters they would like to use with either tool. Starting a Screen or tmux session without these will result in them being terminated with the idle session timeout. The CHPC documentation for these commands will be updated to include the extra details to start a Screen or tmux session in the PE.

Our plan is to put these changes live on all the redwood login nodes tomorrow, September 4, before noon. Existing logins will not be affected, and we will not need to reboot the login nodes. Existing sessions will still be affected by the forced Screen sessions until they are restarted. All sessions, existing and new, will be affected by the idle session termination change. We will send an update email once the change is live.

Please be aware that Open OnDemand and FastX are also effective methods to connect to the cluster resources. We welcome any feedback on this change or any other aspect of the upgraded redwood cluster. Please reach out to us at helpdesk@chpc.utah.edu.