|
View:
New views
20 Messages
—
Rating Filter:
Alert me
|
| < Prev | 1 - 2 | Next > |
|
|
MacPorts AutoBuildI just finished uploading what I've been working on for the last few
weeks off and on, my redo of my old DarwinPorts AutoBuild scripts. See <http://trac.macports.org/wiki/MPAB> Right now it builds ports fine in the chroot (though now with 10.5.3 I need to rebuild a newer version of my chroot), but does have a few limitations. One is that I've currently only tested it with 10.5 and Xcode 3.0. See the Todo section in the ReadMe.txt file that comes with the archive. I've had it successfully build about 145 ports so far, with several failing for various reasons. It starts to slow down when you are dealing when building a port with many dependencies as it will uninstall all ports between each build attempt for better cleanroom building. My MBP's HD gets a bit slow when frequently extracting then removing thousands of files.... Give it a try if you'd like and reply here or update the wiki page. Bryan _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn May 31, 2008, at 5:24 PM, Bryan Blackburn wrote:
> I've had it successfully build about 145 ports so far, with several > failing for various reasons. It starts to slow down when you are > dealing when building a port with many dependencies as it will > uninstall all ports between each build attempt for better cleanroom > building. My MBP's HD gets a bit slow when frequently extracting then > removing thousands of files.... > > Give it a try if you'd like and reply here or update the wiki page. I'm running it now, and it seems pretty cool. I don't know if you have plans for future development, but I was thinking the following would be good: - Add a way to add more macports.conf settings (enabling parallel builds perhaps) - Use archive mode or activate/inactivate to speed things up for ports with lots of dependencies (unless I'm missing something, I think that ports end up getting rebuilt multiple times as dependencies since everything gets uninstalled between port builds) We could also do a 'distributed' build test by having the build app ask a macports server for a port (or list of ports) it should build test and then it could send back the success/failure + log which could be stored. We could then send out nag emails (or open tickets) automatically for ports that don't build, which would be nice. Alternatively, we could set up a dedicated build-test system somewhere that could test ports as they are changed (perhaps a post-commit hook could add to a list of ports to test), but I don't think that macosforge has resources for that (yet?) Thoughts? -- Daniel J. Luke +========================================================+ | *---------------- dluke@... ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+ _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 1:10 PM, Daniel J. Luke wrote:
> - Add a way to add more macports.conf settings (enabling parallel > builds perhaps) > - Use archive mode or activate/inactivate to speed things up for > ports with lots of dependencies (unless I'm missing something, I > think that ports end up getting rebuilt multiple times as > dependencies since everything gets uninstalled between port builds) nevermind, I just noticed that you already have it set up to do this :) -- Daniel J. Luke +========================================================+ | *---------------- dluke@... ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+ _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 10:10 AM, Daniel J. Luke wrote: > On May 31, 2008, at 5:24 PM, Bryan Blackburn wrote: >> I've had it successfully build about 145 ports so far, with several >> failing for various reasons. It starts to slow down when you are >> dealing when building a port with many dependencies as it will >> uninstall all ports between each build attempt for better cleanroom >> building. My MBP's HD gets a bit slow when frequently extracting >> then >> removing thousands of files.... >> >> Give it a try if you'd like and reply here or update the wiki page. > > > Alternatively, we could set up a dedicated build-test system > somewhere that could test ports as they are changed (perhaps a post- > commit hook could add to a list of ports to test), but I don't think > that macosforge has resources for that (yet?) > Once MacPorts has the software ready for a build system, I can start requesting hardware to support it. I have not looked at MPAB yet to see how close it is to an automatic build system. I also remember James saying MPWA would be used as the front-end for such a system, and I dont think MPWA is ready either. So, if someone wants to lead the initiative to assemble the parts and explain how I can deploy it on hardware, I'd be happy to ask for more servers here. If no one wants to take charge of this, I will eventually work on it, but dont know when I'll have time. -Bill --- William Siegrist Mac OS Forge http://macosforge.org/ _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 11:11 AM, William Siegrist wrote: > Once MacPorts has the software ready for a build system, I can start > requesting hardware to support it. I have not looked at MPAB yet to > see how close it is to an automatic build system. I also remember > James saying MPWA would be used as the front-end for such a system, > and I dont think MPWA is ready either. > > So, if someone wants to lead the initiative to assemble the parts > and explain how I can deploy it on hardware, I'd be happy to ask for > more servers here. If no one wants to take charge of this, I will > eventually work on it, but dont know when I'll have time. We have a considerable collection of hardware lying around which could be repurposed to such a role rather easily. There's a machine you use every day for backups, for example, which was, in fact, originally conceived and allocated to this exact purpose (you must have wondered at that attached RAID). Of course that simply never happened, at least not beyond my own primitive attempts, all of which yielded such terrible results [>50% failure rates] that I came to have dark suspicions about my methodology for creating the build chroot (as well as dark suspicions about how many of our ports actually build at any given time) and went on to other things. Anyway, if MPAB/MPWA are starting to generate good results on whatever development hardware is being used, and by "good results" I mean all of the below: o Building, or at least iterating through, all the ports in the system and generating helpful status information for each (even if it's "falls over immediately", that's good to know) o Taking at least some pains to isolate the build products from the builder host machine, both in files read and files written. o Has had all reasonable precautions taken after doing a reasonably pragmatic analysis of the security implications of executing all that open-ended Tcl code and how a "rogue port" might attack the builder, either deliberately or through carelessness. Then I'd say it's time for us to start thinking seriously about putting this into early production. This is the same checklist the project is going to have to go down before anyone will be willing to even "BETA" this on more private hardware anyway, so I don't think it's an unreasonable one. - Jordan _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 11:10 AM, Daniel J. Luke wrote:
... > > I don't know if you have plans for future development, but I was > thinking the following would be good: > > - Add a way to add more macports.conf settings (enabling parallel > builds perhaps) Currently, I simply punt on this issue, since it'd be tough to figure out what settings for various macports.conf values would be best for a given use. You can use './mpab mount' to mount the chroot and 'vi mpchroot/opt/local/etc/macports/macports.conf' to change whatever you need. > > - Use archive mode or activate/inactivate to speed things up for > ports with lots of dependencies (unless I'm missing something, I > think that ports end up getting rebuilt multiple times as > dependencies since everything gets uninstalled between port builds) As you noticed, it does use archive mode, but since it uninstalls all installed ports after each port-build attempt, you'll see it reinstall popular dependencies (eg, pkgconfig, zlib, etc) quite a bit. This way it should catch ports that have missed necessary deps. Fortunately with archive mode it only has to reinstall the files, but if you're on a slower disk, some ports with 20+ deps can still take a while to do these; especially larger ports with lots of files, like ncursesw with over 3000 files alone, seriously IO bound. Bryan > > ... > -- > Daniel J. Luke > +========================================================+ > | *---------------- dluke@... ----------------* | > | *-------------- http://www.geeklair.net -------------* | > +========================================================+ > | Opinions expressed are mine and do not necessarily | > | reflect the opinions of my employer. | > +========================================================+ _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 12:11 PM, William Siegrist wrote:
> ... > > > Once MacPorts has the software ready for a build system, I can start > requesting hardware to support it. I have not looked at MPAB yet to > see how close it is to an automatic build system. I also remember > James saying MPWA would be used as the front-end for such a system, > and I dont think MPWA is ready either. > MPAB works now, though I've only built maybe 5% of MacPorts's ports on my MBP, and even that has generated several trac tickets (and a few I think I forgot to file as well). > So, if someone wants to lead the initiative to assemble the parts > and explain how I can deploy it on hardware, I'd be happy to ask for > more servers here. If no one wants to take charge of this, I will > eventually work on it, but dont know when I'll have time. > If you look at the readme in the tarball, I currently claim the necessary parts are 10.5.[23], Xcode 3.0, and Apple's X11 (including X11SDK from Xcode of course); then a tarball of MacPorts itself. After that, 'sudo ./mpab' should start. It builds in a chroot, so the first run will take some time as it builds the chroot (4.5G or so). Bryan > > -Bill > > > --- > William Siegrist > Mac OS Forge > http://macosforge.org/ > _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 1:28 PM, Jordan K. Hubbard wrote:
> ... > Of course that simply never happened, at > least not beyond my own primitive attempts, all of which yielded such > terrible results [>50% failure rates] that I came to have dark > suspicions about my methodology for creating the build chroot (as well > as dark suspicions about how many of our ports actually build at any > given time) and went on to other things. > While what I've written does have some of it early lineage from your buildall.sh script, it's definitely changed a bit. The good news is that of the 150+ or so ports I had MPAB try to build here, a vast majority of them built successfully. The failures were all explainable at the MacPorts level, not MPAB (eg missing distfiles, a perl module needing 'port -f install', and so on). > Anyway, if MPAB/MPWA are starting to generate good results on whatever > development hardware is being used, and by "good results" I mean all > of the below: > > o Building, or at least iterating through, all the ports in the system > and generating helpful status information for each (even if it's > "falls over immediately", that's good to know) > MPAB currently builds the list of ports to attempt by sorting them by dependency: the early ones it builds are dependencies of later ones. The reason for this is that if Y depends on X and X failed, it'll simply skip Y. Also, of course, if X did build successfully, all other ports depending on it will simply cause X do install from the portarchive instead of rebuilding again. If you extract the MPAB tarball, the chroot-scripts/genportlist.tcl is the script which builds this initial list. > o Taking at least some pains to isolate the build products from the > builder host machine, both in files read and files written. > It's a chroot, so other than the initial building of the chroot, the host machine's files should be left alone. Of course, the bad thing is chroot needs root to run... > o Has had all reasonable precautions taken after doing a reasonably > pragmatic analysis of the security implications of executing all that > open-ended Tcl code and how a "rogue port" might attack the builder, > either deliberately or through carelessness. > The biggest issue here is breaking out of the chroot, otherwise if something malicious happens then all future build attempts would most likely fail quite spectacularly. Bryan > Then I'd say it's time for us to start thinking seriously about > putting this into early production. This is the same checklist the > project is going to have to go down before anyone will be willing to > even "BETA" this on more private hardware anyway, so I don't think > it's an unreasonable one. > > - Jordan > _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 4:47 PM, Bryan Blackburn wrote:
> Currently, I simply punt on this issue, since it'd be tough to figure > out what settings for various macports.conf values would be best for a > given use. You can use './mpab mount' to mount the chroot and 'vi > mpchroot/opt/local/etc/macports/macports.conf' to change whatever you > need. Yeah, I modified where you turned on archivemode to also tweak the conf to do parallel build with some of the cores on my workstation. >> - Use archive mode or activate/inactivate to speed things up for >> ports with lots of dependencies (unless I'm missing something, I >> think that ports end up getting rebuilt multiple times as >> dependencies since everything gets uninstalled between port builds) > > As you noticed, it does use archive mode, but since it uninstalls all > installed ports after each port-build attempt, you'll see it reinstall > popular dependencies (eg, pkgconfig, zlib, etc) quite a bit. This way > it should catch ports that have missed necessary deps. Fortunately > with archive mode it only has to reinstall the files, but if you're on > a slower disk, some ports with 20+ deps can still take a while to do > these; especially larger ports with lots of files, like ncursesw with > over 3000 files alone, seriously IO bound. Have you tried using activate/deactivate instead of archives/ uninstall? I don't know if it would be significantly faster or not, but it would probably be worth testing. I might eventually have time to investigate this (but probably not for the next week or so). -- Daniel J. Luke +========================================================+ | *---------------- dluke@... ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+ _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 2:07 PM, Bryan Blackburn wrote: > While what I've written does have some of it early lineage from your > buildall.sh script, it's definitely changed a bit. The good news is > that of the 150+ or so ports I had MPAB try to build here, a vast > majority of them built successfully. The failures were all > explainable at the MacPorts level, not MPAB (eg missing distfiles, a > perl module needing 'port -f install', and so on). That's excellent - it sounds like you've definitely made much more progress! > MPAB currently builds the list of ports to attempt by sorting them by > dependency: the early ones it builds are dependencies of later ones. > The reason for this is that if Y depends on X and X failed, it'll > simply skip Y. Also, of course, if X did build successfully, all > other ports depending on it will simply cause X do install from the > portarchive instead of rebuilding again. If you extract the MPAB > tarball, the chroot-scripts/genportlist.tcl is the script which builds > this initial list. Does it also support picking up where it left off, or is it an assumption that once you've generated this initial list of ports, you're committed to trying to build the whole thing from beginning to end? The latter scenario is fine, I'm just wondering if there's currently any bookkeeping associated with which ports on that list have been tried and which still need to be built. If there were, it would obviously make it easier for volunteer builders to start and stop batch builds in order to take advantage of slack time on "borrowed" hardware at ${theirInstitution} and we could also discuss the notion of "jobbing out" builds to remote builders as the next logical enhancement. I know, I know, I want everything. :-) > It's a chroot, so other than the initial building of the chroot, the > host machine's files should be left alone. Of course, the bad thing > is chroot needs root to run... Well, until we figure out some way of simulating chroot with some future "trace mode on steroids" I don't see any easy way around this, so I'm just happy you're at least using a chroot! :) > The biggest issue here is breaking out of the chroot, otherwise if > something malicious happens then all future build attempts would most > likely fail quite spectacularly. It's hard enough to break out of a chroot that I'm pretty comfortable with the level of security this represents. It's the trace-mode-on- steroids approach that's a bit more difficult to get right, and I wasn't sure if you had gone that route or not. Where does one check out the current state of MPAB goodness so far again? I must have missed that somewhere in this discussion chain. I've got some unused hardware I wouldn't mind throwing at testing this for awhile... Thanks! - Jordan _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 3:13 PM, Daniel J. Luke wrote:
... > > > Have you tried using activate/deactivate instead of archives/ > uninstall? I don't know if it would be significantly faster or not, > but it would probably be worth testing. I might eventually have time > to investigate this (but probably not for the next week or so). > Hmm, that's an interesting thought; since activate only creates hard links it'll definitely be faster than extracting the archive followed by creating those hard links. While inactive, ports shouldn't be looking in /opt/local/var/macports/software for stuff, so should be safe. Thanks for the idea. Bryan > -- > Daniel J. Luke > +========================================================+ > | *---------------- dluke@... ----------------* | > | *-------------- http://www.geeklair.net -------------* | > +========================================================+ > | Opinions expressed are mine and do not necessarily | > | reflect the opinions of my employer. | > +========================================================+ > _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 4:50 PM, Jordan K. Hubbard wrote:
> ... > > Does it also support picking up where it left off, or is it an > assumption that once you've generated this initial list of ports, > you're committed to trying to build the whole thing from beginning to > end? The latter scenario is fine, I'm just wondering if there's > currently any bookkeeping associated with which ports on that list > have been tried and which still need to be built. If there were, it > would obviously make it easier for volunteer builders to start and > stop batch builds in order to take advantage of slack time on > "borrowed" hardware at ${theirInstitution} and > we could also discuss the notion of "jobbing out" builds to remote > builders as the next logical enhancement. > > I know, I know, I want everything. :-) > Yeah, don't we all? What it does is check to see if there's already a built package (from using archive mode) for the version/revision it's about to build; if there is, it doesn't bother trying to build again. If not (eg, it failed before, or never tried before) then it obviously tries to build. And it does use trap to allow Ctrl-C, moving the pertinent logs out of the chroot and doing cleanup before stopping. That's how I was able to build all those ports initially on a MBP while still using said machine. ... >> > Where does one check out the current state of MPAB goodness so far > again? I must have missed that somewhere in this discussion chain. > I've got some unused hardware I wouldn't mind throwing at testing this > for awhile... > Some basic stuff at <http://trac.macports.org/wiki/MPAB>, the readme in the tarball has more info currently than that wiki page. Bryan > Thanks! > > - Jordan > _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildBryan Blackburn wrote:
> On Jun 20, 2008, at 3:13 PM, Daniel J. Luke wrote: > > ... >> >> Have you tried using activate/deactivate instead of archives/ >> uninstall? I don't know if it would be significantly faster or not, >> but it would probably be worth testing. I might eventually have time >> to investigate this (but probably not for the next week or so). >> > > Hmm, that's an interesting thought; since activate only creates hard > links it'll definitely be faster than extracting the archive followed > by creating those hard links. While inactive, ports shouldn't be > looking in /opt/local/var/macports/software for stuff, so should be > safe. Thanks for the idea. I assume you're using trunk, so it should be fine, but I just thought I'd mention that it won't work with 1.6.0 because of #7361: <http://trac.macports.org/ticket/7361>. - Josh _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 20, 2008, at 8:00 PM, Bryan Blackburn wrote:
>> Have you tried using activate/deactivate instead of archives/ >> uninstall? I don't know if it would be significantly faster or not, >> but it would probably be worth testing. I might eventually have time >> to investigate this (but probably not for the next week or so). > > Hmm, that's an interesting thought; since activate only creates hard > links it'll definitely be faster than extracting the archive followed > by creating those hard links. While inactive, ports shouldn't be > looking in /opt/local/var/macports/software for stuff, so should be > safe. Thanks for the idea. using direct mode instead of image mode would be an easy minor performance improvement (saving the time used to make the hardlinks). Probably not as much of a win as being able to use activate/ deactivate, though. -- Daniel J. Luke +========================================================+ | *---------------- dluke@... ----------------* | | *-------------- http://www.geeklair.net -------------* | +========================================================+ | Opinions expressed are mine and do not necessarily | | reflect the opinions of my employer. | +========================================================+ _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn Jun 22, 2008, at 8:46 AM, Daniel J. Luke wrote:
> On Jun 20, 2008, at 8:00 PM, Bryan Blackburn wrote: >> >> Hmm, that's an interesting thought; since activate only creates hard >> links it'll definitely be faster than extracting the archive followed >> by creating those hard links. While inactive, ports shouldn't be >> looking in /opt/local/var/macports/software for stuff, so should be >> safe. Thanks for the idea. > > If it doesn't work for some reason (like #7361 or some other issue) > using direct mode instead of image mode would be an easy minor > performance improvement (saving the time used to make the > hardlinks). Probably not as much of a win as being able to use > activate/deactivate, though. > So far it's working fine, it's tried to build about 170 ports (23 failed, some because of dependencies failing). Watching things like the chroot/opt/local/[lib|bin] show things coming and going as they should, which should allow ports with undeclared dependencies to fail when not finding what they want. I didn't do any stopwatch runs before/after so I can't say if things are any better, but since I've usually been sitting at my machine when much of the build is going, it definitely feels like it's getting through more ports. Bryan > -- > Daniel J. Luke > +========================================================+ > | *---------------- dluke@... ----------------* | > | *-------------- http://www.geeklair.net -------------* | > +========================================================+ > | Opinions expressed are mine and do not necessarily | > | reflect the opinions of my employer. | > +========================================================+ _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuildOn May 31, 2008, at 16:24, Bryan Blackburn wrote: > I just finished uploading what I've been working on for the last few > weeks off and on, my redo of my old DarwinPorts AutoBuild scripts. > See > > <http://trac.macports.org/wiki/MPAB> The page says MPAB is to be downloaded from the attachment on that wiki page. Why not put it in the repository somewhere, like in /users/ blb? > Right now it builds ports fine in the chroot (though now with 10.5.3 I > need to rebuild a newer version of my chroot), but does have a few > limitations. One is that I've currently only tested it with 10.5 and > Xcode 3.0. See the Todo section in the ReadMe.txt file that comes > with the archive. > > I've had it successfully build about 145 ports so far, with several > failing for various reasons. It starts to slow down when you are > dealing when building a port with many dependencies as it will > uninstall all ports between each build attempt for better cleanroom > building. My MBP's HD gets a bit slow when frequently extracting then > removing thousands of files.... > > Give it a try if you'd like and reply here or update the wiki page. Does MPAB just try to build the port with its default variants, or does it try all permutations of variants the port will accept? I imagine it's not uncommon for maintainers to test a new version of a port with the default variants but forget (or not have the time) to test all variants, so variants might break over time. Patches only added in variants might need to be updated. Some variants might conflict with one another but the maintainer may not have marked that. Etc. _______________________________________________ macports-dev mailing list macports-dev@... http://lists.macosforge.org/mailman/listinfo.cgi/macports-dev |
|
|
Re: MacPorts AutoBuild |