Alright, was my program, namely the following:
tcgetattr(STDIN_FILENO, &terminalSettings);
struct termios termSet = terminalSettings;
atexit([]{ // Reset at exit
tcsetattr(STDIN_FILENO, TCSAFLUSH, &terminalSettings);
});
termSet.c_lflag &= ~ECHO;
termSet.c_lflag &= ~ICANON;
termSet.c_cc[VMIN] = 0;
termSet.c_cc[VTIME] = 0;
tcsetattr(STDIN_FILENO, TCSAFLUSH, &termSet);
which I set up at some point to correctly receive input in the terminal as well as over SSH.
Though there is one other part later that also triggers the same, so I'll have to search more tomorrow.