IRC chat logs for #ltsp on irc.libera.chat (webchat)
Channel log from 24 October 2021 (all times are UTC)
| 07:34 | woernie has joined IRC (woernie!~werner@p5b296c2c.dip0.t-ipconnect.de) | |
| 08:40 | Vercas has left IRC (Vercas!~Vercas@gateway/tor-sasl/vercas, Remote host closed the connection) | |
| 08:41 | Vercas has joined IRC (Vercas!~Vercas@gateway/tor-sasl/vercas) | |
| 10:37 | Vercas has left IRC (Vercas!~Vercas@gateway/tor-sasl/vercas, Remote host closed the connection) | |
| 10:37 | ricotz has joined IRC (ricotz!~ricotz@ubuntu/member/ricotz) | |
| 10:37 | Vercas has joined IRC (Vercas!~Vercas@gateway/tor-sasl/vercas) | |
| 12:05 | woernie has left IRC (woernie!~werner@p5b296c2c.dip0.t-ipconnect.de, Remote host closed the connection) | |
| 12:54 | woernie has joined IRC (woernie!~werner@p5b296c2c.dip0.t-ipconnect.de) | |
| 13:16 | Vercas has left IRC (Vercas!~Vercas@gateway/tor-sasl/vercas, Remote host closed the connection) | |
| 13:16 | Vercas has joined IRC (Vercas!~Vercas@gateway/tor-sasl/vercas) | |
| 13:55 | shored has joined IRC (shored!~shored@user/shored) | |
| 14:14 | eu^ip-74-84-236- has joined IRC (eu^ip-74-84-236-!~eu^ip-74-@74.84.236.83) | |
| 14:14 | eu^ip-74-84-236- is now known as Monkberry | |
| 14:16 | <Monkberry> Hey Alkis are you around?
| |
| 14:18 | I ran some updates and some stuff broke. I was hoping you could help.
| |
| 14:20 | <alkisg> Hi Monkberry
| |
| 14:20 | <Monkberry> Hi
| |
| 14:20 | <alkisg> What broke?
| |
| 14:21 | <Monkberry> Epoptes apparently. I've seen some posts of your about issues with it but none of them seem to fit
| |
| 14:21 | <alkisg> What happens, e,g, it won't start, the clients don't appear, etc?
| |
| 14:21 | <Monkberry> Auto login appears to have broke as well for clients on the LTSP platform
| |
| 14:22 | shored has left IRC (shored!~shored@user/shored, Quit: ZNC 1.8.2+deb2 - https://znc.in) | |
| 14:22 | <Monkberry> I can't look at them or control them as well as not start some of them, all of that worked before the updates
| |
| 14:23 | <alkisg> Do you want me to check over VNC? I'll need vnc from both your server and a client
| |
| 14:23 | <Monkberry> On ones that I can start, I can access a root terminal on them via epoptes or I can ssh into them directly
| |
| 14:23 | <alkisg> (or a VM, if you have one, I don't recall)
| |
| 14:23 | x11vnc -connect alkisg.ltsp.org
| |
| 14:28 | Monkberry: you have two failed services, unifi and ua-messaging
| |
| 14:28 | unifi seems to delay ltsp client startup enough so that lightdm doesn't autologin
| |
| 14:28 | <Monkberry> I've been watching
| |
| 14:28 | <alkisg> Let's start by fixing them on the server first
| |
| 14:28 | <Monkberry> ok
| |
| 14:28 | <alkisg> Do you know why it fails? Do you need it, should I fix or remove it?
| |
| 14:29 | I'm guessing it's some kind of service for unifi routers?
| |
| 14:30 | <Monkberry> unifi doesn't even need to run on that box
| |
| 14:31 | You could just remove it, I forgot all about that package
| |
| 14:31 | <alkisg> OK removing
| |
| 14:32 | <Monkberry> ok
| |
| 14:32 | <alkisg> mongodb claims that it's not used by anything else; removing
| |
| 14:35 | I doubt canonical gives linuxmint support via ubuntu-advantage
| |
| 14:35 | Shall I remove that as well?
| |
| 14:36 | Nah many things depend on it; I just masked it
| |
| 14:37 | How easy is it to reboot the server, just to check that it has a clean systemctl status?
| |
| 14:37 | If it's a big deal, we can skip it
| |
| 14:40 | <Monkberry> We can reboot it
| |
| 14:42 | I'm not there, I'm about an hour away but we can reboot it when that image rebuilds. I also wanted to check and see if I could remotely boot some of the other clients with epoptes after
| |
| 14:42 | I saw you remove a bunch of kernels and stuff. Was all this down to that unifi package?
| |
| 14:44 | <alkisg> No, these were packages that you have removed but not purged
| |
| 14:44 | So I just removed their "state", nothing important
| |
| 14:45 | If the clients have been shut down without the "wol state" in the NIC, they don't listen for wol anymore
| |
| 14:45 | Lets check if autologin works first
| |
| 14:45 | <Monkberry> i know but they were (some that I know of) working yesterday before the update
| |
| 14:45 | ok
| |
| 14:46 | should we run "ltsp initrd"?
| |
| 14:46 | before the reboot that is
| |
| 14:47 | I see that you rebooted the ltsp client, I thought you meant the server
| |
| 14:51 | <alkisg> Let's make the client autologin first
| |
| 14:55 | Monkberry: so, let's assume that unifi started having troubles and is causing systemd to go into a failed state, and the "network-online" target to not be reached
| |
| 14:55 | That means that epoptes-client won't launch, and the NIC won't be put into the "wol mode"
| |
| 14:55 | That then means that even if we fixed it now, you still need to manually poweron the clients
| |
| 14:55 | <Monkberry> Would that explain why lightdm is not starting?
| |
| 14:56 | <alkisg> one time only; then they'll get the wol state
| |
| 14:56 | No
| |
| 14:56 | <Monkberry> I see that the auto login works after a manual reboot of the lightdm service
| |
| 14:57 | ah wait a minute now
| |
| 14:57 | <alkisg> Did you somehow disable lightdm?
| |
| 14:57 | <Monkberry> I did manually disable lightdm on the server because someone accidentally logged into it and shut it down
| |
| 14:57 | <alkisg> Ah, then the clients would also inherit that
| |
| 14:58 | <Monkberry> So I disabled lightdm on the server so they couldn't login to it anymore
| |
| 14:58 | ah
| |
| 14:58 | <alkisg> OK, we'll do it like this:
| |
| 14:58 | <Monkberry> now I'm feeling pretty dumb
| |
| 14:58 | <alkisg> the server will reach console.target, not multi-user.target
| |
| 14:58 | While lightdm will be enabled
| |
| 14:58 | <Monkberry> ah perfect
| |
| 14:58 | <alkisg> *sorry,I meant graphical vs multi-user, I'll google it :D
| |
| 14:58 | OK doing so
| |
| 14:59 | Hmm, how did you disable lightdm?
| |
| 15:00 | <Monkberry> The server sits in a room where they take the kids if they are not feeling well. etc. Last week one of the teachers went in there, saw the login screen and logged in and then turned it completely off thinking it was just another workstation
| |
| 15:00 | systemctl disable lightdm
| |
| 15:01 | <alkisg> And yet `systemctl enable lightdm` shows some errors... hmmm...
| |
| 15:01 | <Monkberry> So me thinking about an easy fix, running that command above just booted the server to a prompt, where if I was there and wanted a gui, I just logged in and ran startx
| |
| 15:02 | <alkisg> How do you login to the server remotely, e.g. via xrdp, ssh + vnc?
| |
| 15:02 | <Monkberry> idk maybe the package got removed or borked in some way
| |
| 15:02 | If I'm not there, I use x2go via ssh
| |
| 15:03 | <alkisg> How are you connected now? I don't see x2go running
| |
| 15:03 | <Monkberry> via x2go
| |
| 15:04 | <alkisg> Sorry I misttyped x2g0
| |
| 15:04 | zero instead of oh :)
| |
| 15:04 | <Monkberry> I'm in Lily Dale, about an hour away (on crappy dsl connection but it's working amazingly fast)
| |
| 15:04 | ah
| |
| 15:06 | For the lightdm thing, an "apt-get install --reinstall lightdm" might have fixed it
| |
| 15:06 | This is too cool, I'm in Lily Dale, you're in Greece and I'm watching you work on a server in Kenmore
| |
| 15:07 | <alkisg> Haha, and actually I"m at home, remoting to my office PC where the reverse vnc connection happens
| |
| 15:08 | <Monkberry> lol
| |
| 15:09 | You should probably select /dev/sda with the space bar
| |
| 15:12 | <alkisg> Yeah I guess it was cloned from another disk
| |
| 15:12 | So... systemctl enable lightdm didn't do it, so as you said I reinstalled it,
| |
| 15:12 | I set the server's default target to multi-user, which does start x2go,
| |
| 15:12 | <Monkberry> I saw that, yes I clone disks all the time, to save time
| |
| 15:12 | <alkisg> (but not lightdm),
| |
| 15:12 | while the clients remain with target = graphical, which starts lightdm
| |
| 15:13 | So when you want to "startx" on the server, do this instead:
| |
| 15:13 | sudo systemctl isolate graphical.target.
| |
| 15:13 | Without the dot: sudo systemctl isolate graphical.target
| |
| 15:13 | <Monkberry> Will I need to do something different to use x2go?
| |
| 15:14 | <alkisg> Theoretically no
| |
| 15:14 | It should be the same
| |
| 15:14 | <Monkberry> ok cool
| |
| 15:16 | <alkisg> Do you have anyone remotely to press the poweron button at the clients, just one time?
| |
| 15:17 | <Monkberry> we could try with first floor room 104 or room 105
| |
| 15:18 | If need be, I could get someone over there
| |
| 15:20 | <alkisg> I think the problem is fixed, but someone will need to boot them one time, and then WOL will work after that
| |
| 15:20 | Now the network cards are sleeping without listening to WOLs
| |
| 15:21 | Because lightdm or unifi blocked epoptes-client from running and from enabling their WOLs
| |
| 15:21 | We can poweroff the TV and test if we can wake it up after that :)
| |
| 15:21 | <Monkberry> I will have someone there in 10 min
| |
| 15:22 | I noticed before that I could remotely start pcs that were on the LTSP system but not others
| |
| 15:23 | <alkisg> If you install epoptes-client in the others, you'll be able to wol them too
| |
| 15:23 | <Monkberry> I just tried booting another off the LTSP server
| |
| 15:23 | <alkisg> Or if you go to network-manager properties and enable wol in the connection settings
| |
| 15:23 | <Monkberry> I did and it used to work
| |
| 15:24 | The server did shut down last night around 8pm for no apparent reason as well
| |
| 15:26 | The client pc we have been working on should be no difference to any of the others there, except for the distinction that it is an LTSP client
| |
| 15:27 | <alkisg> I'm not sure I got what you're saying; do you think there's still a problem that we haven't solved?
| |
| 15:27 | <Monkberry> not sure
| |
| 15:28 | <alkisg> If so, have someone boot a PC that is affected, so that we can look at its WOL settings
| |
| 15:28 | <Monkberry> They should be there in 5 min
| |
| 15:28 | <alkisg> ethtool eth0 => Wake-on: g
| |
| 15:28 | <Monkberry> I told him we need to boot the pcs in room 104 and 105
| |
| 15:28 | <alkisg> While if it's like this, it's disabled: Wake-on: d
| |
| 15:29 | It would be better to boot them all, then power them off via epoptes
| |
| 15:29 | <Monkberry> Yes and they were all working before, except that they might have not gotten shut down correctly or that the server went down before they did
| |
| 15:29 | <alkisg> Then they'll be listening for wol events, and you'll be able to wake them up at anytime
| |
| 15:30 | <Monkberry> i know, my guess is that they went down AFTER the server went down, or didn't go down at all.
| |
| 15:30 | and they're now just hanging there
| |
| 15:30 | <alkisg> If you reboot the ltsp server while the ltsp clients are running, they do reconnect in epoptes
| |
| 15:31 | Also, wol is set at boot, so it wouldn't matter if the ltsp server went down while they were up
| |
| 15:31 | <Monkberry> interesting
| |
| 15:31 | I've never seen that happen, usually they just hang with some garble on the screen
| |
| 15:32 | <alkisg> In the old ltsp, they hanged
| |
| 15:32 | In the new ltsp, /home hangs (so the user session hangs), while / continues
| |
| 15:32 | But you have nfs, so /home shouldn't hang either
| |
| 15:32 | <Monkberry> ah that makes sense
| |
| 15:32 | <alkisg> It should just continue where it left off
| |
| 15:33 | <Monkberry> if /home hangs. it would appear the whole thing hangs
| |
| 15:33 | <alkisg> nbd and sshfs hang, nfs continues
| |
| 15:33 | <Monkberry> the homes are nfs mounted
| |
| 15:33 | <alkisg> Old ltsp = nbd and sshfs; new ltsp = nfs and sshfs; your ltsp = nfs and nfs
| |
| 15:33 | So nothing should hang
| |
| 15:33 | <Monkberry> well the whole thing really
| |
| 15:34 | <alkisg> We can test if you want, we can reboot the server while TV is up
| |
| 15:34 | <Monkberry> that nbd was a NIGHTMARE
| |
| 15:34 | ok
| |
| 15:34 | <alkisg> You do have a static ip on the server, right?
| |
| 15:34 | <Monkberry> yes
| |
| 15:34 | <alkisg> OK, if you want, reboot the server
| |
| 15:34 | <Monkberry> ok
| |
| 15:34 | <alkisg> Hopefully we'll be able to reconnect via x2go etc :)
| |
| 15:35 | <Monkberry> ah yes
| |
| 15:35 | that wouldn't be good if I couldn't get in somehow
| |
| 15:37 | Did you install grub to /dev/sda?
| |
| 15:37 | ah ok hang on
| |
| 15:38 | <alkisg> Yes, although it shouldn't matter it was already there
| |
| 15:39 | See, everything is perfect :D
| |
| 15:39 | nfs ==> real file system! :D
| |
| 15:39 | <Monkberry> The camera viewer (mvp) was running and then it stopped
| |
| 15:41 | <alkisg> Maybe it dies if it can't write to disk for a minute
| |
| 15:41 | That's an app issue though
| |
| 15:41 | <Monkberry> I just tried to use epoptes to control the pc and I can't login
| |
| 15:41 | try it
| |
| 15:42 | oh now it worked
| |
| 15:44 | it seems to work (sometimes) or with a very long delay
| |
| 15:45 | <alkisg> I think that's a bug in the new epoptes code
| |
| 15:45 | I haven't pinpointed it, let me have a look
| |
| 15:45 | <Monkberry> xtigervncviewer?
| |
| 15:46 | <alkisg> That part is ok; what is not ok is that it's trying to run [xtigervncviewer,
| |
| 15:47 | With [ and comma
| |
| 15:47 | The weird thing is that it does this SOME times only, so I haven't been able yet to see what goes wrong
| |
| 15:49 | <Monkberry> maybe a clue, I just had someone over there and manually started the pcs in 104 and 105 and we see nothing on epoptes
| |
| 15:49 | I could ssh into those if needed
| |
| 15:49 | <alkisg> Sure yes please
| |
| 15:49 | <Monkberry> ok
| |
| 15:49 | <alkisg> I don't think it's related to epoptes though
| |
| 15:54 | I don't see any dhcp requests etc after the ltsp server has been booted
| |
| 15:54 | Let's reboot the TV pc to check if it boots
| |
| 15:56 | <Monkberry> ok, I'm logged into both pcs now, 104 and 105
| |
| 15:56 | <alkisg> $ epoptes-client
| |
| 15:56 | Epoptes-client connecting to server:789... ...done
| |
| 15:56 | epoptes-client: 382: [xtigervncviewer,: not found
| |
| 15:56 | <Monkberry> hang on and I'll give you shells on them
| |
| 15:56 | <alkisg> Thanks
| |
| 16:04 | Monkberry: there's some firewall involved somewhere
| |
| 16:04 | <Monkberry> Should not be anything on the inside
| |
| 16:04 | I can reboot the firewall if you like
| |
| 16:05 | it's pfsense and it is the dhcp server as well
| |
| 16:08 | <alkisg> Hmm no
| |
| 16:08 | The problem is somehow related to the previous epoptes certificate
| |
| 16:08 | By generating a new one, the problem is solved,
| |
| 16:09 | but then, you'll need to run `epoptes-client -c` in all your clients...
| |
| 16:09 | Is that acceptable?
| |
| 16:09 | <Monkberry> room 104 is a stand alone install (not ltsp) but it does mount it's home via nfs
| |
| 16:09 | <alkisg> You'll need to ssh to everyone of them, and run epoptes-client -c
| |
| 16:09 | Then epoptes-client will work
| |
| 16:10 | <Monkberry> I can do that
| |
| 16:10 | <alkisg> If that's OK, we can stop debugging, problem solved
| |
| 16:10 | OK, can you also ssh to the TV ltsp client?
| |
| 16:10 | <Monkberry> I do appreciate all your help and I'll take care of you
| |
| 16:10 | <alkisg> Because that one now doesn't have the correct certificate
| |
| 16:10 | <Monkberry> sure hang on
| |
| 16:11 | <alkisg> There's still the problem with the [xtighervncviewer', but that one's on me :)
| |
| 16:11 | I'll try to find out when it happens
| |
| 16:11 | <Monkberry> That TV pc DOES boot via epoptes though, that must be because it is on LTSP?
| |
| 16:12 | I have noticed that (at least it used to be), that the wol needed to be initiated from .11 if the PC was on .11 (old vlan stuff)
| |
| 16:12 | however, unless the pc is on the .10 side, it would NOT boot into the LTSP server (again, old vlan stuff)
| |
| 16:12 | <alkisg> That part is because you can't relay WOLs easily
| |
| 16:16 | What's the IP for .105?
| |
| 16:16 | *Room 105
| |
| 16:17 | <Monkberry> 172.16.10.61 I believe or .11.61
| |
| 16:17 | <alkisg> OK ssh there so that we fix that one as well
| |
| 16:17 | It's just an `epoptes-client -c; systemctl restart epoptes`
| |
| 16:18 | <Monkberry> no it's 172.16.11.11
| |
| 16:19 | <alkisg> OK that's it. Btw you can use the hostname.local instead of the ip.
| |
| 16:19 | Actually give me a few minutes to check the tigervnc issue
| |
| 16:19 | <Monkberry> ok I can fix the rest up now, thank you very much! What did the issue appear to be? Multiple I know
| |
| 16:19 | ok
| |
| 16:20 | <alkisg> The main issue was that you disabled lightdm
| |
| 16:20 | The epoptes certificate issue, I've no idea what caused it, e.g. expiry or certificate chains or cipher mismatch
| |
| 16:21 | <Monkberry> ah
| |
| 16:21 | <alkisg> While this new xtigervnc issue is on me, it started with the latest patches a couple of weeks ago
| |
| 16:21 | From what I can see, it fails one time, it succeeds the next; very weird
| |
| 16:21 | <Monkberry> one of those perfect storm scenarios
| |
| 16:22 | <alkisg> Btw realvnc-vnc-viewer is a lot better than tigervnc and supports automatic scaling, if you want to switch to that
| |
| 16:22 | <Monkberry> Yes, I would love to
| |
| 16:23 | <alkisg> (x2go somehow freezes for a while when vnc'ing to the tv pc)
| |
| 16:23 | <Monkberry> On the clients before, I could ctrl-enter and it would unmaximize (usually their windows vm) and I could then access their linux desktop
| |
| 16:23 | or ctrl-alt-enter
| |
| 16:23 | <alkisg> Can you still operate the screen via x2go?
| |
| 16:24 | It's frozen here
| |
| 16:24 | <Monkberry> yes, it looks like you got kicked out
| |
| 16:24 | <alkisg> x11vnc -connect alkisg.ltsp.org
| |
| 16:24 | <Monkberry> I'll vnc you again
| |
| 16:31 | <alkisg> Monkberry: please close the tv-pc VNC window
| |
| 16:31 | It freezed my vnc again :D
| |
| 16:31 | Thanks
| |
| 16:32 | Close it again please :)
| |
| 16:35 | : -listen, 52217] 52217
| |
| 16:35 | I think python somehow fails to use the array properly half of the times :(
| |
| 16:38 | Monkberry: please close it again
| |
| 16:38 | OK ready I found it
| |
| 16:38 | It's partly my fault, partly python's
| |
| 16:38 | I'm using the same variable name, cmd, in two scopes
| |
| 16:39 | And python sometimes sees the wrong scope
| |
| 16:39 | I'll just rename the var
| |
| 16:39 | <Monkberry> ah very good
| |
| 16:39 | <alkisg> It's fixed now for you, but I'll also send an update for all the other users
| |
| 16:39 | We can close VNC now, good work for both of us! :)
| |
| 16:40 | <Monkberry> Thank you Alkisg! Do I still have to ssh to all the clients and rerun epoptes-client -c?
| |
| 16:40 | <alkisg> Yes
| |
| 16:40 | (except for ltsp clients of course)
| |
| 16:40 | <Monkberry> ok no biggie
| |
| 16:40 | <alkisg> We solved 3 separate issues today!
| |
| 16:40 | <Monkberry> did you change the vnc server?
| |
| 16:41 | Is it back to where I can ctrl-alt-enter?
| |
| 16:42 | <alkisg> Ah no I forgot about that
| |
| 16:44 | Monkberry: my kid needs me for something, and you have many pcs; let's do this instead:
| |
| 16:44 | Please read our new documentation about vnc viewers: https://epoptes.org/documentation/vnc/
| |
| 16:45 | Try to apply it yourself (the realvnc-vnc-viewer is the best; the xvnc4viewer the second-best; the ssvnc the third-best)
| |
| 16:45 | And if you can't make it work for some reason, I can help you another time
| |
| 16:46 | <Monkberry> ok sounds good Alkisg! Thank you again and I'll paypal ya.
| |
| 16:46 | <alkisg> Thank you too! :)
| |
| 16:47 | <Monkberry> Now, if I can figure out how to print this conversation out.....
| |
| 16:55 | I figured it out, have a good one!
| |
| 16:55 | Monkberry has left IRC (Monkberry!~eu^ip-74-@74.84.236.83, Quit: Client closed) | |
| 18:31 | vagrantc has joined IRC (vagrantc!~vagrant@2600:3c01:e000:21:21:21:0:100b) | |
| 18:53 | woernie has left IRC (woernie!~werner@p5b296c2c.dip0.t-ipconnect.de, Remote host closed the connection) | |
| 21:35 | ricotz has left IRC (ricotz!~ricotz@ubuntu/member/ricotz, Quit: Leaving) | |
| 23:25 | vagrantc has left IRC (vagrantc!~vagrant@2600:3c01:e000:21:21:21:0:100b, Quit: leaving) | |
| 23:56 | lucas__ has joined IRC (lucas__!~lucascast@189.90.44.253.jupiter.com.br) | |
| 23:59 | lucas_ has left IRC (lucas_!~lucascast@177-185-128-123.static.isotelco.net.br, Ping timeout: 264 seconds) | |