IRC chat logs for #ltsp on irc.libera.chat (webchat)
Channel log from 9 November 2022 (all times are UTC)
| 00:51 | vagrantc has left IRC (vagrantc!~vagrant@2600:3c01:e000:21:7:77:0:20, Quit: leaving) | |
| 00:54 | we6jbo has joined IRC (we6jbo!~we6jbo@cpe-66-75-235-109.san.res.rr.com) | |
| 03:48 | vagrantc has joined IRC (vagrantc!~vagrant@2600:3c01:e000:21:7:77:0:20) | |
| 04:10 | shored has left IRC (shored!~shored@user/shored, Ping timeout: 246 seconds) | |
| 04:10 | shored has joined IRC (shored!~shored@user/shored) | |
| 04:51 | we6jbo has left IRC (we6jbo!~we6jbo@cpe-66-75-235-109.san.res.rr.com, Remote host closed the connection) | |
| 06:16 | ricotz has joined IRC (ricotz!~ricotz@ubuntu/member/ricotz) | |
| 06:17 | woernie has joined IRC (woernie!~werner@p5b2962ea.dip0.t-ipconnect.de) | |
| 06:31 | vagrantc has left IRC (vagrantc!~vagrant@2600:3c01:e000:21:7:77:0:20, Quit: leaving) | |
| 09:02 | parlos has joined IRC (parlos!~pal@nplgateway.nplab.bth.se) | |
| 10:36 | lliurex^SERVERIN has joined IRC (lliurex^SERVERIN!~lliurex^S@213.0.87.24) | |
| 10:36 | lliurex^SERVERIN has left IRC (lliurex^SERVERIN!~lliurex^S@213.0.87.24, Client Quit) | |
| 14:08 | alkisg_irc1 has joined IRC (alkisg_irc1!~Thunderbi@2a02:587:7458:8100:c24a:ff:fe02:bc1e) | |
| 14:09 | alkisg_irc has left IRC (alkisg_irc!~Thunderbi@srv1-dide.ioa.sch.gr, Ping timeout: 248 seconds) | |
| 14:10 | alkisg_irc1 is now known as alkisg_irc | |
| 15:31 | oh207_ has joined IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 15:34 | bluejaypop4 is now known as bluejaypop | |
| 16:19 | <joris> I am preparing an LTSP server. So if I understand it correct clients can save documents on their user accounts. So that would make it advisable to have a large ssd hosting the images?
| |
| 16:20 | <alkisg> Hi joris , you need a large ssd for /home, not for /srv/ltsp/images
| |
| 16:20 | The clients boot using a single networked image, and then they connect to the server /home via sshfs, and all the user files go there, not inside the image
| |
| 16:22 | <joris> ah ok cool thanks for explaining alkisg
| |
| 17:18 | Monkberry has joined IRC (Monkberry!~Monkberry@static-108-34-16-131.bflony.fios.verizon.net) | |
| 17:18 | <Monkberry> alkisg are you busy?
| |
| 17:19 | <alkisg> Hi Monkberry , I'm OK, what's up?
| |
| 17:19 | <Monkberry> Hi. I've a strange problem and it seems to be between ltsp and pulseaudio
| |
| 17:20 | <alkisg> What are the symptoms?
| |
| 17:20 | <Monkberry> the clients that boot off of ltsp, sometimes have sound, sometimes not
| |
| 17:20 | I can see (in the client) multiple instances of pulseaudio running
| |
| 17:20 | if I kill them all and restart pulseaudio, all is well
| |
| 17:20 | <alkisg> Is that on Ubuntu MATE 22.04?
| |
| 17:20 | vagrantc has joined IRC (vagrantc!~vagrant@2600:3c01:e000:21:7:77:0:20) | |
| 17:21 | <Monkberry> mate something, like 20 ish I believe
| |
| 17:21 | I'm creating the image off the root of the system with "ltsp image /"
| |
| 17:21 | <alkisg> Do you have a broken client right now so that I can see the `ps faux` list?
| |
| 17:22 | <Monkberry> the one that was broke I fixed so idk
| |
| 17:22 | today that is
| |
| 17:22 | <alkisg> At some point you'll need to upgrade. Upgrading might, or might not, fix the issue. So maybe start with that?
| |
| 17:22 | <Monkberry> its at that school again
| |
| 17:22 | <alkisg> You can also keep a backup (while it's running) of the old OS to revert if something goes wrong
| |
| 17:23 | <Monkberry> it's the middle of the day here and they're all pounding away
| |
| 17:23 | I saw something like NOT including the person in the audio group but that didn't work
| |
| 17:23 | I was hoping it was something that needed to be excluded from making the image
| |
| 17:24 | <alkisg> No I haven't heard this issue before, it's unique to you
| |
| 17:24 | <Monkberry> figures lol
| |
| 17:24 | <alkisg> When it happens again, do keep the `ps faux > somefile` output from that affected client,
| |
| 17:24 | <Monkberry> ok will do
| |
| 17:25 | <alkisg> we'll see then why multiple pulseaudios are running. Also, consider if you want to upgrade to 22.04, which might solve the issue, saving us the troubleshooting process :)
| |
| 17:25 | In 22.04 it's possible to switch to pipewire, although it's not the default
| |
| 17:26 | <Monkberry> i've been afraid to upgrade that system, it would be a lot of work if it gets borked
| |
| 17:29 | <alkisg> 22.04 is very similar to 20.04, I didn't have any upgrade issues except for some ldap installations that needed a pwpolicy upgrade
| |
| 17:29 | But a backup would certainly be needed to be able to revert easily in case of issues
| |
| 17:29 | Let's start with the `ps faux` output when it happens again, and we'll see after that...
| |
| 17:30 | <Monkberry> ok thanks
| |
| 17:49 | how would you like me to send the file(s)?
| |
| 17:50 | <alkisg> Monkberry: you may email me if you wish
| |
| 17:54 | <Monkberry> done
| |
| 17:59 | <alkisg> Monkberry: this ps faux is from the client?
| |
| 18:00 | It seems like it's not doing proper logoffs then; I can see more than 5 user names running processes there
| |
| 18:02 | pamltsp is supposed to instruct systemd to kill all process on logout:
| |
| 18:02 | ltsp/client/login/pamltsp: sed 's/.*\(KillUserProcesses=\).*/\1yes/' -i /etc/systemd/logind.conf
| |
| 18:02 | Do you have anything weird regardings homes and authentication, or are you using the default sshfs?
| |
| 18:03 | Finally, I only see a single pulseaudio process there
| |
| 18:03 | The point was to see if the other pulseaudios were from the same user or not, and when they were spawned, e.g. in a previous login or in that same login
| |
| 18:43 | <Monkberry> i know, by that time, I fixed the pulseaudio thing, at least for today
| |
| 18:43 | I did see all the other processes owned by other people, I thought that was definitely wrong. I'm just using all the stock default ssh stuff as it was
| |
| 18:49 | and yes, that was run from the client
| |
| 19:01 | <alkisg> Check if the problem only happens when they quickly logout/relogin without waiting for 10 seconds for systemd to kill their old login
| |
| 19:02 | <Monkberry> I'm rebooting the one with the sound issue now. No, I don't believe anyone would have done what you thought above
| |
| 19:03 | All the homes are owned by whomever they should be
| |
| 19:04 | I just rebooted the person that had the sound issue, logged in via epoptes and no sound card shows at all
| |
| 19:09 | I sent you another email
| |
| 19:14 | and now another email
| |
| 19:15 | <alkisg> Monkberry: you've installed something that runs on boot and logs in as all these users
| |
| 19:15 | <Monkberry> i've now idea what that might be
| |
| 19:15 | <alkisg> A service. E.g. maybe something related to postfix? What did you install that pulled postfix as a dependency?
| |
| 19:15 | <Monkberry> i've no idea what that might be
| |
| 19:16 | idk, how would I look?
| |
| 19:16 | <alkisg> Want to vnc so that I have a look on an unused client?
| |
| 19:16 | !vnc-edide
| |
| 19:16 | <ltspbot> vnc-edide: To share your screen with me, open Epoptes → Help menu → Remote support → Host: srv1-dide.ioa.sch.gr, and click the Connect button
| |
| 19:16 | <Monkberry> ok, let's do that
| |
| 19:21 | <alkisg> Monkberry: the same logins also happen on the server
| |
| 19:21 | And it's strange that e.g. there's an ed login, but not a room123 login
| |
| 19:21 | What's different between these accounts?
| |
| 19:21 | <Monkberry> There is no room123
| |
| 19:21 | <alkisg> It's an example
| |
| 19:21 | <Monkberry> and nobody should be logging onto the server
| |
| 19:21 | <alkisg> I wonder if linuxmint has something very broken there
| |
| 19:22 | Yes, I don't mean physically. I mean that you have a broken software that logs them in
| |
| 19:22 | <Monkberry> just an fyi, I did get a ups disconnect notice when I rebooted ed
| |
| 19:22 | idk
| |
| 19:23 | woernie has left IRC (woernie!~werner@p5b2962ea.dip0.t-ipconnect.de, Remote host closed the connection) | |
| 19:23 | <Monkberry> That's why I initially thought something was needed in the exclude file for the image creation, idk
| |
| 19:23 | <alkisg> Nov 08 21:02:38 standrews-ltsp systemd[816]: pam_unix(systemd-user:session): session opened for user ed2 by (uid=0)
| |
| 19:23 | No the problem isn't related to ltsp. That login is from your server, a root process logged in ed2
| |
| 19:24 | Let me check crontabs...
| |
| 19:24 | <Monkberry> that one is weird, I did create a user ed2 when I thought maybe his profile was borked as I looked into this sound issue but ed2 has not logged in in a long time now
| |
| 19:24 | <alkisg> These accounts do NOT log in physically
| |
| 19:24 | A software does that without you knowing about it
| |
| 19:25 | <Monkberry> ok
| |
| 19:25 | <alkisg> And we need to pinpoint the software and remove it, because it autostarts pulseaudio for everyone in all ltsp clients
| |
| 19:25 | I ran sudo -i, type the password please
| |
| 19:25 | <Monkberry> i gathered that as I sent you that screenshot I took
| |
| 19:25 | ok
| |
| 19:28 | <alkisg> Monkberry: I want to do some changes in ltsp.conf and reboot a test client
| |
| 19:28 | Which one can I use?
| |
| 19:28 | <Monkberry> it looks like michelle is gone for the day, use that one
| |
| 19:28 | <alkisg> OK
| |
| 19:32 | oh207 has joined IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 19:32 | oh207_ has left IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 248 seconds) | |
| 19:42 | oh207 has left IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 260 seconds) | |
| 19:43 | <alkisg> I'm starting to think it's related to your /etc/skel; you put a lot of stuff in there, and it has the names of the autostarted logins...
| |
| 19:44 | <Monkberry> ok
| |
| 19:45 | I use that because if I create a lot of users it's easy and fast to "useradd -m -k /etc/skel"
| |
| 19:46 | This way, I can edit one user, put their home as /etc/skel and create all the others as to look and feel the same
| |
| 19:49 | <alkisg> It's not the correct way to do it as then all users will have their settings path as /home/templateuser
| |
| 19:49 | But anyway that's a different topic...
| |
| 19:49 | <Monkberry> I edit the /etc/passwd file
| |
| 19:50 | <alkisg> The settings are saved in binary databases such as gsettings dconf and mozilla sqlite
| |
| 19:50 | <Monkberry> ah
| |
| 19:50 | then what is /etc/skel even there for?
| |
| 19:51 | of is this only ltsp affected?
| |
| 19:52 | <alkisg> It's there for the gold old programs that don't use the path in their settings, e.g. .bashrc or .profile
| |
| 19:52 | This issue is related to multiuser systems; it's completely unrelated to ltsp
| |
| 19:52 | <Monkberry> ah
| |
| 19:52 | <alkisg> I'm trying to find a reason that e.g. ed2 is autologged in, and room111 isn't
| |
| 19:52 | What's different between these accounts...
| |
| 19:53 | <Monkberry> ed2 was/uses ltsp, room111 has their own install/hdd
| |
| 19:53 | <alkisg> You're talking about the physical room111 PC, while I'm talking about the room111 user account on the ltsp server
| |
| 19:53 | <Monkberry> I have the rooms 111 mount their homes via nfs, at least that's the way it was during covid, same as it is now but that's why it started out that way
| |
| 19:54 | I created the room users so they could have id's the same as on the server so they could remotely mount their homes
| |
| 19:55 | via nfs
| |
| 19:56 | <alkisg> I think the difference is that ed2 is a member of vboxusers
| |
| 19:56 | Looking at that weird "vbox-autostart" service now; maybe that's the one to blame
| |
| 19:57 | E.g. maybe it tries to automatically start some VMs that these users have, and runs pulseaudio as a side effect
| |
| 19:57 | <Monkberry> I was also going to use ed2 as remoting in from home, none of them need to be in the vbox users group on the ltsp server
| |
| 19:58 | i have an entirely different box that runs all their vms
| |
| 20:00 | that vbox group on the ltsp server can be gone entirely, that was my bad, probably got installed during some, let's do this moment", who knows now
| |
| 20:10 | <alkisg> I can't find what logs in these users (systemd slices). I'll ask at the #systemd channel in case someone knows
| |
| 20:11 | <Monkberry> I've not seen the image rebuilt. Do you want me to yank virtualbox and then rebuild the image?
| |
| 20:12 | <alkisg> No
| |
| 20:12 | I'm hunting possible ideas
| |
| 20:12 | <Monkberry> ok
| |
| 20:12 | <alkisg> But none of them worked yet
| |
| 20:12 | I'm not sure if the issue is virtualbox,postfix or something else
| |
| 20:12 | <Monkberry> i've been just watching
| |
| 20:13 | just an fyi, postfix has probably been on that box for a long time, idk when virtualbox got installed
| |
| 20:13 | <alkisg> Monkberry: if you're planning to remove virtualbox anyway, we can do that, while we're waiting for answers from the #systemd channel
| |
| 20:13 | <Monkberry> ok, I'll do that now
| |
| 20:14 | <alkisg> Hmm, timeshift, is that for the users too?
| |
| 20:15 | Or just for the root file system?
| |
| 20:15 | <Monkberry> idk about that one
| |
| 20:15 | I didn't implicedly install that
| |
| 20:16 | The system is currently locked with apt, pid 158774
| |
| 20:17 | root 158774 158771 1 15:14 pts/4 00:00:02 /usr/bin/apt full-upgrade
| |
| 20:18 | I didn't start this
| |
| 20:18 | it looks like you did, I'll wait it out then
| |
| 20:19 | <alkisg> Yes I thought you were watching :)
| |
| 20:19 | I found out the problem
| |
| 20:19 | It's "systemd lingering"
| |
| 20:19 | Either you or some dummy linuxmint setting enabled lingering for these users
| |
| 20:20 | I.e. to autostart their processes
| |
| 20:20 | <Monkberry> I wouldn't even know how
| |
| 20:20 | does the server need a reboot?
| |
| 20:22 | oh207 has joined IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 20:22 | <alkisg> POST_INIT_RM_LINGER="rm -f /var/lib/systemd/linger/*"
| |
| 20:22 | This already fixed it without removing virtualbox or running ltsp image
| |
| 20:23 | The server doesn't need to be rebooted, no
| |
| 20:23 | <Monkberry> ok
| |
| 20:23 | <alkisg> You want me to remove virtualbox and run ltsp image anyway?
| |
| 20:23 | <Monkberry> I was trying to remember why it's even on there in the first place
| |
| 20:24 | It can be removed, I don't see any vm running
| |
| 20:24 | yes
| |
| 20:24 | <alkisg> OK
| |
| 20:24 | <Monkberry> please
| |
| 20:27 | oh207 has left IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 260 seconds) | |
| 20:27 | oh207_ has joined IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 20:28 | <Monkberry> That linger thing was part of a kernel thing?
| |
| 20:29 | <alkisg> No. Something configured systemd to "autostart" the processes of these users
| |
| 20:29 | <Monkberry> hmmm
| |
| 20:30 | my experience has always been, if you don't know why, it'll happen again
| |
| 20:30 | <alkisg> Then we should keep that line in ltsp.conf
| |
| 20:30 | <Monkberry> absolutely
| |
| 20:30 | <alkisg> It's possible that this is a default thing for linuxmint, and anyone that logs in on the server, gets that setting from them and onward
| |
| 20:30 | So if these accounts have logged in on the server, that could be it
| |
| 20:31 | (that would make it a linuxmint bug; I haven't seen that thing on ubuntu)
| |
| 20:34 | <Monkberry> I really doubt any of those users would have physically logged into that computer. This is the server you fixed that had the weird graphics thing that a school teacher sat down at, thought it was just another workstation and shut the thing off when she was done so we disabled the gui just on that box but not on the created image? remember that one?
| |
| 20:34 | <alkisg> See room109. Lingering there was enabled in 2020 for carrie, emma and peter
| |
| 20:35 | Maybe you had thin client mode enabled, and they logged in to their workstations, and that effectively logged them in to the server?
| |
| 20:35 | Or maybe you were testing something with their accounts on the server?
| |
| 20:35 | <Monkberry> that box, would have it's own hdd. They used to leave the kids in all the same rooms and the teachers would go from room to room, hence why I initially made the mounted (roaming) homes
| |
| 20:36 | oh207 has joined IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 20:36 | <Monkberry> idk
| |
| 20:36 | oh207_ has left IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 260 seconds) | |
| 20:36 | <alkisg> In all my boxes, /var/lib/systemd/linger is empty; that's definitely a linuxmint issue
| |
| 20:36 | <Monkberry> hmmm
| |
| 20:37 | I don't recall how I used to add users but it has varied
| |
| 20:38 | <alkisg> Ouch I didn't notice you had run ltsp image today
| |
| 20:38 | ltsp image remembers the current and the previous image
| |
| 20:38 | Which means that the old-old image is now gone
| |
| 20:38 | Which means any clients that were booted before Nove 9 10:40 are now hanged
| |
| 20:38 | ...and need to be rebooted
| |
| 20:39 | They might be able to save their /homes though as that's via sshfs
| |
| 20:39 | <Monkberry> I've only ever seen that on systems where the server rebooted and the workstation was using it
| |
| 20:40 | The day is nearly over now so I don
| |
| 20:40 | 't think it'll be a big deal
| |
| 20:40 | I've just rebooted Ed as it was his sound that started this whole thing
| |
| 20:40 | <alkisg> OK. It's possible that they won't even reboot, they'll hand on shutdown, unless you type `sudo reboot -f` from epoptes
| |
| 20:42 | <Monkberry> Ed was on ltsp completely and I just rebooted him all ok
| |
| 20:42 | and his sound appears to work now (it did before on occassion and then wouldn't)
| |
| 20:43 | I did ask him if it seemed like a time thing, if he came in later would he not have sound, etc
| |
| 20:43 | I was starting to suspect that it was a pecking order or daemon amount issue
| |
| 20:43 | but I knew that I should never see all those processes owned by other people
| |
| 20:44 | <alkisg> I'm certain we did solve the "multiple logins / pulseaudio processes" issue, and I'm 95% sure that this was causing the sound issue
| |
| 20:44 | I think we can leave it at that now, and you can ping me if you see any related issues in the future
| |
| 20:44 | <Monkberry> that's great alkisg! Thank you as always and I'll send you a gift for your help, thanks again!
| |
| 20:45 | <alkisg> Thank you, cheers :)
| |
| 20:46 | Monkberry has left IRC (Monkberry!~Monkberry@static-108-34-16-131.bflony.fios.verizon.net, Quit: Client closed) | |
| 20:48 | oh207_ has joined IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 20:48 | oh207 has left IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 260 seconds) | |
| 21:13 | ricotz has left IRC (ricotz!~ricotz@ubuntu/member/ricotz, Quit: Leaving) | |
| 21:30 | oh207 has joined IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 21:31 | oh207_ has left IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 260 seconds) | |
| 22:03 | oh207_ has joined IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 22:03 | oh207 has left IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 252 seconds) | |
| 22:22 | oh207 has joined IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net) | |
| 22:23 | oh207_ has left IRC (oh207_!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 260 seconds) | |
| 23:31 | oh207 has left IRC (oh207!~oh207@pool-72-69-11-48.nycmny.fios.verizon.net, Ping timeout: 248 seconds) | |