428124 - mac buildbot slaves should reboot ready for use

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Splitting out from bug#417887, as each o.s. will have different gotchas.

Basically, how to make each buildbot master/slaves reboot cleanly, reconnect and handle new jobs?

Comment 1

[[e-mail] (:bhearsum)]

IIRC we don't have a way to launch Buildbot properly on boot in OS X. I believe there is hdiutil security context problems when it is launched from a startup script. This will require further investigation.

Comment 2

[Rob Campbell [:rc] (:robcee)]

yeah, the problem on mac is that you need to be on the console when you launch the process so you inherit security settings from the login window.

We might be able to fake that by invoking an AppleScript from the user's StartupItems that does something like the following:

tell application "Terminal"
	do script "buildbot start /builds/slave"
end tell

Comment 3

[Rob Campbell [:rc] (:robcee)]

taking this to test out on qm-moz2mini01.

Comment 4

[[e-mail] (:bhearsum)]

We chatted a bunch about this today and decided that part of this will be doing scheduled, periodic reboots of staging machines both to iron out kinks in the rebooting and to look for potential performance gains.

Comment 5

[[e-mail] (:bhearsum)]

Thanks to the work in bug 430833 this was easy-peasy to do on moz2-darwin9-slave08. I ran it overnight and all builds went green, except for mozilla-central leak tests, which are legitimately busted right now. I'm going to roll this out on the rest of the staging Macs later today. Once that's run for a bit we can roll it out in production.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: enable periodic reboots on staging for macs

Comment 7

[[e-mail] (:bhearsum)]

Attachment: buildbot launch agent file

Comment 8

[[e-mail] (:bhearsum)]

I should probably document the details here:
* Ensure /builds/slave is the slavedir or a symlink to it
* Download buildbot.start.slave.plist and put it in /Library/LaunchAgents
* Make sure it is owned by root:wheel

Copy and paste:
sudo wget --no-check-certificate -Obuildbot.start.slave.plist https://bug428124.bugzilla.mozilla.org/attachment.cgi?id=361795
sudo chown root:wheel buildbot.start.slave.plist 

From VNC:
* Make sure the resolution is set to 1280x1024.
* System Prefs -> Accounts -> Login Options
** Set 'Automatic Login' to 'cltbld', enter the password when prompted.

Reboot.

Comment 9

[[e-mail] (:bhearsum)]

Comment on attachment 361784 [details] [diff] [review]
enable periodic reboots on staging for macs

changeset:   938:9710768473dd

Comment 10

[[e-mail] (:bhearsum)]

Alright, these changes have been deployed on moz2-darwin9-slave03, 04, and 08 and periodic reboots of them have been enabled in staging. Once we're confident they're stable this can be rolled out to production.

Comment 11

[[e-mail] (:bhearsum)]

I found out this morning that Macs need an /etc/sudoers update just like Linux. I added this to the staging Macs, hopefully they'll reboot this time.

Comment 12

[[e-mail] (:bhearsum)]

Just had the first Mac reboot and come back up successfully. Let's keep running this for awhile to make sure it's stable.

Comment 13

[[e-mail] (:bhearsum)]

Some of the Macs haven't been coming back up properly. I've added the following to the plist file to try and help with that:
<key>StartInterval</key>
<integer>600</integer>

This will cause OS X to try and start it every 600 seconds, which will fail gracefully if it's already started but _should_ start it up if it happens not to upon boot.

Comment 14

[[e-mail] (:bhearsum)]

I haven't seen a Mac fail to restart its slave since I added the <key>StartInterval</key> parameter. This is ready to deploy.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: updated plist file

Here's an updated plist file with the StartInterval in it.

Comment 16

[[e-mail] (:bhearsum)]

This is getting deployed tomorrow. Here's copy/paste instructions (assumes /builds/slave is the slavedir or a symlink to it):

cd /Library/LaunchAgents
sudo wget --no-check-certificate -Obuildbot.start.slave.plist https://bugzilla.mozilla.org/attachment.cgi?id=364362
sudo chown root:wheel buildbot.start.slave.plist 



And once the slave becomes idle it should be rebooted to ensure everything got installed okay.

Comment 17

[[e-mail] (:bhearsum)]

Attachment: plist, correct syntax

Here's the plist file with the correct syntax. I didn't have time to deploy it on everything today, but the following slaves are setup for clean boots:
bm-xserve05
bm-xserve04
bm-xserve16
bm-xserve17
bm-xserve18
bm-xserve19
bm-xserve20
bm-xserve21
bm-xserve22
fx-mac-1.9-slave1
fx-mac-1.9-slave2
moz2-darwin9-slave01
moz2-darwin9-slave02
moz2-darwin9-slave03
moz2-darwin9-slave04
moz2-darwin9-slave05
moz2-darwin9-slave06
moz2-darwin9-slave07
moz2-darwin9-slave08

This only slaves the try slaves (try-mac-slave01 -> try-mac-slave05), which I will finish up with on Monday.

Comment 18

[Nick Thomas [:nthomas] (UTC+12)]

Adjusting summary to remove masters, since they all run on linux.

Comment 19

[[e-mail] (:bhearsum)]

Alright, I've deployed this changes (with the CVS_RSH key) to the try server pool: try-mac-slave01 -> try-mac-slave05. I'm still waiting for slave04 to become idle to make sure it comes back up okay. Things are looking good other than that.

Still need to update support docs and inventory, almost done here though.

Comment 20

[[e-mail] (:bhearsum)]

Whoops, messed up the dependencies.

Comment 21

[[e-mail] (:bhearsum)]

Okay, this has been deployed on all of our Mac buildbot slaves now, and the support docs have been updated. We're all done here.

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Also fixed up the CVS_RSH=ssh on fx--mac-1.9-slave1 & 2.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

And on bm-xserve16 thru 19 & 22, moz2-darwin9-slave01 thru 08 - need CVS for release update verify.

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Final plist (with CVS_RSH=ssh)