|
»
»
»
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
|
View:
New views
11 Messages
—
Rating Filter:
Alert me
|
 |
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by Ian Rogers
::
Rate this Message:
Steve Blackburn wrote:
> On 23/04/2008, at 1:25 PM, Ian Rogers wrote: > >> Steve Blackburn wrote: >> >>> My feeling is that we have to be very careful not to tune for the >>> benchmarks. >>> >> :-) I thought that was the point of benchmarks. >> > > Yikes! > > Benchmarks exist to give us an imperfect indication of how our system > might perform in the real world. We want to tune to the real world, > not to specific benchmarks. Yikes, we've got non-real world benchmarks! ;-) > Of course benchmarks are often very > useful in identifying performance problems and we should always use > them to do that, but we should not be making changes to the VM just so > that it will perform on a particular benchmark (unless of course we > believe that the performance issue tickled by that benchmark > represents an important real world scenario). > Have you an example of a performance issue that is tickled by a benchmark but doesn't occur in the real world? This sounds like a bug in the benchmark. >> Ah, changing the benchmark we profile didn't actually change the >> code.. >> > > I agree. But as Dave said, it didn't solve the problem either :-) > Ah, I'll blame other unclosed trackers for that. We should be recomputing the method summary based on opt compiler experience. Having poor coverage isn't going to benefit the situation. >> I think the problem here is that the profiled-image benchmark isn't >> giving us good coverage of the libraries and the RVM. For example, fop >> has very little exception code in it. >> > > Absolutely. It is there more or less by accident. > It is the shortest DaCapo benchmark to run, so it's a good choice to see a build isn't broken :-) >> Maybe the best option is to run >> every DaCapo benchmark once. >> > > Sure, but that takes a lot of time. > I'm more worried we'll end up with no decent profiling information, convert all our branches to conditional moves and push performance backwards. >> It'd be nice if DaCapo was much less Java 1.4, then when I do >> boxing/unboxing improvements I can see a kink in the performance >> graphs :-) >> > > Sure. It is an open source suite. If you have any suggestions, we're > all ears.... ....specially if you're prepared to help out with the non- > trivial task of evaluation and packaging (it generally takes weeks to > months of full time work to get a benchmark ready---eclipse took me > over a month, I think). > I'll have to package PearColator up ;-) Ian >> Having just sat through many papers at ISPASS on correlating the >> similarities between different benchmarks, maybe a similar analysis >> can >> be applied to DaCapo. >> > > See the paper. We did PCA analysis for exactly this reason. > > >> One problem with eclipse is that it covers too >> much territory, indeed the most of any of our regular benchmarks. This >> can mean we lose any useful profile details in the weight of data we >> get. >> > > We try to include a mix of workloads in DaCapo. I'm working on a much > bigger beast right now. Big complex benchmarks are an important piece > of the puzzle. > > Micro benchmarks also have a very important diagnostic role, but we > did not focus on that in DaCapo. > > --Steve > > > >> Ian >> >> >>> On 23/04/2008, at 12:49 PM, Ian Rogers wrote: >>> >>> >>> >>>> Hi Steve, >>>> >>>> we could really do with analyzing all the inlining heuristics with >>>> every >>>> DaCapo benchmark and using the optimal set of benchmarks, >>>> heuristics, >>>> etc. From a grep of DaCapo's source, eclipse doesn't show as using >>>> java.util.Vector - but this could be because it's compressed... >>>> RVM-446 >>>> is a nuisance. We'd like to fix the precise type setting of this >>>> operands for final classes for 2.9.3, but this has broken the >>>> compress >>>> inlining making it too aggressive and regressed its performance. >>>> This is >>>> gating the 2.9.3 release. The intended fixes haven't improved >>>> situation >>>> and we've also now regressed db. A problem is the age of SpecJVM and >>>> its >>>> short execution time - meaning simple inlining decisions can be >>>> key to >>>> making sure we don't cause register allocator difficulties and >>>> costing >>>> us 10s of percent of performance. >>>> >>>> Maybe other core team members are aware of/have done work in this >>>> area? >>>> It seems like a natural target for machine learning techniques. >>>> >>>> Regards, >>>> Ian >>>> >>>> Steve Blackburn wrote: >>>> >>>> >>>>> Hi Ian, >>>>> >>>>> We should give some thought to this. I threw in fop as the profile >>>>> benchmark after very little consideration. If we're prepared to >>>>> pay >>>>> the build-time overhead, I suggest we use a richer benchmark, >>>>> such as >>>>> eclipse which is more likely to give the whole VM a shake. I chose >>>>> fop mainly because it was short running.... What do you (and >>>>> others) >>>>> think? >>>>> >>>>> I suppose one of us could actually do an evaluation of this and see >>>>> how each of the options fly... It should not be too hard. I'm >>>>> prepared to do it, but not at this moment. >>>>> >>>>> --Steve >>>>> >>>>> On 23/04/2008, at 12:12 PM, captain5050@... >>>>> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Revision: 14144 >>>>>> http://jikesrvm.svn.sourceforge.net/jikesrvm/?rev=14144&view=rev >>>>>> Author: captain5050 >>>>>> Date: 2008-04-22 19:12:39 -0700 (Tue, 22 Apr 2008) >>>>>> >>>>>> Log Message: >>>>>> ----------- >>>>>> Switch from DaCapo fop to lusearch for profiled image runs. This >>>>>> is >>>>>> an attempt to bump the use of java.util.Vector to increase >>>>>> SpecJVM's >>>>>> db performance. See RVM-446. >>>>>> >>>>>> Modified Paths: >>>>>> -------------- >>>>>> rvmroot/trunk/build.xml >>>>>> >>>>>> Modified: rvmroot/trunk/build.xml >>>>>> = >>>>>> ================================================================== >>>>>> --- rvmroot/trunk/build.xml 2008-04-22 15:41:35 UTC (rev 14143) >>>>>> +++ rvmroot/trunk/build.xml 2008-04-23 02:12:39 UTC (rev 14144) >>>>>> @@ -1548,7 +1548,7 @@ >>>>>> <arg value="-X:base:edge_counters=true"/> >>>>>> <arg value="-X:base:edge_counter_file=${build.profiles}/ >>>>>> profile.ec"/> >>>>>> <arg value="-X:aos:final_report_level=2"/> >>>>>> - <arg line="-jar ${dacapo.jar} -n 3 fop"/> >>>>>> + <arg line="-jar ${dacapo.jar} -n 3 lusearch"/> >>>>>> </exec> >>>>>> </target> >>>>>> >>>>>> >>>>>> >>>>>> This was sent by the SourceForge.net collaborative development >>>>>> platform, the world's largest Open Source development site. >>>>>> >>>>>> ------------------------------------------------------------------------- >>>>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>>>> Don't miss this year's exciting event. There's still time to save >>>>>> $100. >>>>>> Use priority code J8TL2D2. >>>>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>>>>> _______________________________________________ >>>>>> Jikesrvm-commits mailing list >>>>>> Jikesrvm-commits@... >>>>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-commits >>>>>> >>>>>> >>>>>> >>>>> ------------------------------------------------------------------------- >>>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>>> Don't miss this year's exciting event. There's still time to save >>>>> $100. >>>>> Use priority code J8TL2D2. >>>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>>>> _______________________________________________ >>>>> Jikesrvm-core mailing list >>>>> Jikesrvm-core@... >>>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >>>>> >>>>> >>>>> >>>> ------------------------------------------------------------------------- >>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>> Don't miss this year's exciting event. There's still time to save >>>> $100. >>>> Use priority code J8TL2D2. >>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>>> _______________________________________________ >>>> Jikesrvm-core mailing list >>>> Jikesrvm-core@... >>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >>>> >>>> >>> ------------------------------------------------------------------------- >>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>> Don't miss this year's exciting event. There's still time to save >>> $100. >>> Use priority code J8TL2D2. >>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>> _______________________________________________ >>> Jikesrvm-core mailing list >>> Jikesrvm-core@... >>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >>> >>> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >> Don't miss this year's exciting event. There's still time to save >> $100. >> Use priority code J8TL2D2. >> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >> _______________________________________________ >> Jikesrvm-core mailing list >> Jikesrvm-core@... >> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >> > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > Jikesrvm-core mailing list > Jikesrvm-core@... > https://lists.sourceforge.net/lists/listinfo/jikesrvm-core > ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by Ian Rogers
::
Rate this Message:
Steve Blackburn wrote:
> On 23/04/2008, at 8:21 AM, Ian Rogers wrote: > >>> Of course benchmarks are often very >>> useful in identifying performance problems and we should always use >>> them to do that, but we should not be making changes to the VM just >>> so >>> that it will perform on a particular benchmark (unless of course we >>> believe that the performance issue tickled by that benchmark >>> represents an important real world scenario). >>> >> Have you an example of a performance issue that is tickled by a >> benchmark but doesn't occur in the real world? >> > > I didn't say "doesn't occur". I said "represents an important real > world scenario". _209_db is a very well known example. > As in _209_db is overly trivial? > There's a nice paper dedicated to the infamous "health" benchmark. http://doi.acm.org/10.1145/503205.503206 > IIRC this was down to bugs in the benchmark. I believe the performance reports avoid buggy benchmarks. Of course any problems in this area should have trackers and we should be endeavoring to make our performance monitoring process as near to perfect as we can. Where perfect is the same as the real-world, however we choose to define that. Related-issue: one problem I think the RVM has in <=2.9.2 is that in the runtime we leant on the generational collector being good. This meant we weren't aggressive in removing objects that in other VMs would have been in a different heap from the application's (or more likely stack allocated). I've cleaned a lot of this up, but it will have GC implications as previous benchmark results would have been including the overhead of these mechanisms somewhat. >> This sounds like a bug in >> the benchmark. >> > > Indeed. Unfortunately benchmarks are inherently "buggy" (insofar as > they are not perfect representations of the real world). Some > benchmarks are more so than others.... > > This is why individual benchmark results are very poor indicators of > the pros or cons of any system, and why we have suites which (attempt > to) cover much more ground. > Right, I don't think anyone would ever propose adding an optimization that benefited a single benchmark and regressed all others. >>>> Maybe the best option is to run >>>> every DaCapo benchmark once. >>>> >>>> >>> Sure, but that takes a lot of time. >>> >>> >> I'm more worried we'll end up with no decent profiling information, >> convert all our branches to conditional moves and push performance >> backwards. >> > > Perhaps the best option is to generalize the interface, because > clearly it depends on what it is you're trying to do. > I'm not sure I follow. You mean build different profiled images for different uses? What are the different uses other than best common case real-world performance? Ian ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
|
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by David P Grove
::
Rate this Message:
> We should give some thought to this. I threw in fop as the profile > benchmark after very little consideration. If we're prepared to pay > the build-time overhead, I suggest we use a richer benchmark, such as > eclipse which is more likely to give the whole VM a shake. I chose > fop mainly because it was short running.... What do you (and others) > think? > > I suppose one of us could actually do an evaluation of this and see > how each of the options fly... It should not be too hard. I'm > prepared to do it, but not at this moment. > I'd think (without actually measuring anything....;)) that eclipse would be the best choice among the DaCapo benchmarks. What we are looking for is something that exercises a fair amount of the VM codebase in "typical" ways. --dave ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by Ian Rogers
::
Rate this Message:
Hi Steve,
we could really do with analyzing all the inlining heuristics with every DaCapo benchmark and using the optimal set of benchmarks, heuristics, etc. From a grep of DaCapo's source, eclipse doesn't show as using java.util.Vector - but this could be because it's compressed... RVM-446 is a nuisance. We'd like to fix the precise type setting of this operands for final classes for 2.9.3, but this has broken the compress inlining making it too aggressive and regressed its performance. This is gating the 2.9.3 release. The intended fixes haven't improved situation and we've also now regressed db. A problem is the age of SpecJVM and its short execution time - meaning simple inlining decisions can be key to making sure we don't cause register allocator difficulties and costing us 10s of percent of performance. Maybe other core team members are aware of/have done work in this area? It seems like a natural target for machine learning techniques. Regards, Ian Steve Blackburn wrote: > Hi Ian, > > We should give some thought to this. I threw in fop as the profile > benchmark after very little consideration. If we're prepared to pay > the build-time overhead, I suggest we use a richer benchmark, such as > eclipse which is more likely to give the whole VM a shake. I chose > fop mainly because it was short running.... What do you (and others) > think? > > I suppose one of us could actually do an evaluation of this and see > how each of the options fly... It should not be too hard. I'm > prepared to do it, but not at this moment. > > --Steve > > On 23/04/2008, at 12:12 PM, captain5050@... wrote: > > >> Revision: 14144 >> http://jikesrvm.svn.sourceforge.net/jikesrvm/?rev=14144&view=rev >> Author: captain5050 >> Date: 2008-04-22 19:12:39 -0700 (Tue, 22 Apr 2008) >> >> Log Message: >> ----------- >> Switch from DaCapo fop to lusearch for profiled image runs. This is >> an attempt to bump the use of java.util.Vector to increase SpecJVM's >> db performance. See RVM-446. >> >> Modified Paths: >> -------------- >> rvmroot/trunk/build.xml >> >> Modified: rvmroot/trunk/build.xml >> =================================================================== >> --- rvmroot/trunk/build.xml 2008-04-22 15:41:35 UTC (rev 14143) >> +++ rvmroot/trunk/build.xml 2008-04-23 02:12:39 UTC (rev 14144) >> @@ -1548,7 +1548,7 @@ >> <arg value="-X:base:edge_counters=true"/> >> <arg value="-X:base:edge_counter_file=${build.profiles}/ >> profile.ec"/> >> <arg value="-X:aos:final_report_level=2"/> >> - <arg line="-jar ${dacapo.jar} -n 3 fop"/> >> + <arg line="-jar ${dacapo.jar} -n 3 lusearch"/> >> </exec> >> </target> >> >> >> >> This was sent by the SourceForge.net collaborative development >> platform, the world's largest Open Source development site. >> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >> Don't miss this year's exciting event. There's still time to save >> $100. >> Use priority code J8TL2D2. >> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >> _______________________________________________ >> Jikesrvm-commits mailing list >> Jikesrvm-commits@... >> https://lists.sourceforge.net/lists/listinfo/jikesrvm-commits >> > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > Jikesrvm-core mailing list > Jikesrvm-core@... > https://lists.sourceforge.net/lists/listinfo/jikesrvm-core > ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by Steve Blackburn
::
Rate this Message:
My feeling is that we have to be very careful not to tune for the
benchmarks. Of course we always want to improve them, but hopefully we can do that with generic fixes and performance improvements rather than bm-specific fixes. One of the goals of the dacapo suite was that it would change and directly combat the tune-for-benchmark game that goes on. In the background with some of my few spare cycles, I'm working on the next dacapo release. Of course it will have one or two new benchmarks and will drop one or two, but it will also update every one of the existing ones to the latest releases. This means that any hard work to tune to those specific benchmarks is in jeopardy, which of course was our goal :-) My feeling about eclipse was that it might be good because it covers so much territory and thus might give us a well rounded profile. But I don't have any specific data to back this up, just lots of measurements of various aspects of the benchmark which you can find in the dacapo paper (the longer tech report adds jvm98 to the set). --Steve On 23/04/2008, at 12:49 PM, Ian Rogers wrote: > Hi Steve, > > we could really do with analyzing all the inlining heuristics with > every > DaCapo benchmark and using the optimal set of benchmarks, heuristics, > etc. From a grep of DaCapo's source, eclipse doesn't show as using > java.util.Vector - but this could be because it's compressed... > RVM-446 > is a nuisance. We'd like to fix the precise type setting of this > operands for final classes for 2.9.3, but this has broken the compress > inlining making it too aggressive and regressed its performance. > This is > gating the 2.9.3 release. The intended fixes haven't improved > situation > and we've also now regressed db. A problem is the age of SpecJVM and > its > short execution time - meaning simple inlining decisions can be key to > making sure we don't cause register allocator difficulties and costing > us 10s of percent of performance. > > Maybe other core team members are aware of/have done work in this > area? > It seems like a natural target for machine learning techniques. > > Regards, > Ian > > Steve Blackburn wrote: >> Hi Ian, >> >> We should give some thought to this. I threw in fop as the profile >> benchmark after very little consideration. If we're prepared to pay >> the build-time overhead, I suggest we use a richer benchmark, such as >> eclipse which is more likely to give the whole VM a shake. I chose >> fop mainly because it was short running.... What do you (and >> others) >> think? >> >> I suppose one of us could actually do an evaluation of this and see >> how each of the options fly... It should not be too hard. I'm >> prepared to do it, but not at this moment. >> >> --Steve >> >> On 23/04/2008, at 12:12 PM, captain5050@... wrote: >> >> >>> Revision: 14144 >>> http://jikesrvm.svn.sourceforge.net/jikesrvm/?rev=14144&view=rev >>> Author: captain5050 >>> Date: 2008-04-22 19:12:39 -0700 (Tue, 22 Apr 2008) >>> >>> Log Message: >>> ----------- >>> Switch from DaCapo fop to lusearch for profiled image runs. This is >>> an attempt to bump the use of java.util.Vector to increase SpecJVM's >>> db performance. See RVM-446. >>> >>> Modified Paths: >>> -------------- >>> rvmroot/trunk/build.xml >>> >>> Modified: rvmroot/trunk/build.xml >>> =================================================================== >>> --- rvmroot/trunk/build.xml 2008-04-22 15:41:35 UTC (rev 14143) >>> +++ rvmroot/trunk/build.xml 2008-04-23 02:12:39 UTC (rev 14144) >>> @@ -1548,7 +1548,7 @@ >>> <arg value="-X:base:edge_counters=true"/> >>> <arg value="-X:base:edge_counter_file=${build.profiles}/ >>> profile.ec"/> >>> <arg value="-X:aos:final_report_level=2"/> >>> - <arg line="-jar ${dacapo.jar} -n 3 fop"/> >>> + <arg line="-jar ${dacapo.jar} -n 3 lusearch"/> >>> </exec> >>> </target> >>> >>> >>> >>> This was sent by the SourceForge.net collaborative development >>> platform, the world's largest Open Source development site. >>> >>> ------------------------------------------------------------------------- >>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>> Don't miss this year's exciting event. There's still time to save >>> $100. >>> Use priority code J8TL2D2. >>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>> _______________________________________________ >>> Jikesrvm-commits mailing list >>> Jikesrvm-commits@... >>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-commits >>> >> >> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >> Don't miss this year's exciting event. There's still time to save >> $100. >> Use priority code J8TL2D2. >> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >> _______________________________________________ >> Jikesrvm-core mailing list >> Jikesrvm-core@... >> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >> > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save > $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > Jikesrvm-core mailing list > Jikesrvm-core@... > https://lists.sourceforge.net/lists/listinfo/jikesrvm-core ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by Ian Rogers
::
Rate this Message:
Steve Blackburn wrote:
> My feeling is that we have to be very careful not to tune for the > benchmarks. :-) I thought that was the point of benchmarks. > Of course we always want to improve them, but hopefully > we can do that with generic fixes and performance improvements rather > than bm-specific fixes. > Ah, changing the benchmark we profile didn't actually change the code.. I think the problem here is that the profiled-image benchmark isn't giving us good coverage of the libraries and the RVM. For example, fop has very little exception code in it. Maybe the best option is to run every DaCapo benchmark once. > One of the goals of the dacapo suite was that it would change and > directly combat the tune-for-benchmark game that goes on. In the > background with some of my few spare cycles, I'm working on the next > dacapo release. Of course it will have one or two new benchmarks and > will drop one or two, but it will also update every one of the > existing ones to the latest releases. This means that any hard work > to tune to those specific benchmarks is in jeopardy, which of course > was our goal :-) > It'd be nice if DaCapo was much less Java 1.4, then when I do boxing/unboxing improvements I can see a kink in the performance graphs :-) > My feeling about eclipse was that it might be good because it covers > so much territory and thus might give us a well rounded profile. But > I don't have any specific data to back this up, just lots of > measurements of various aspects of the benchmark which you can find in > the dacapo paper (the longer tech report adds jvm98 to the set). > Having just sat through many papers at ISPASS on correlating the similarities between different benchmarks, maybe a similar analysis can be applied to DaCapo. One problem with eclipse is that it covers too much territory, indeed the most of any of our regular benchmarks. This can mean we lose any useful profile details in the weight of data we get. Ian > On 23/04/2008, at 12:49 PM, Ian Rogers wrote: > > >> Hi Steve, >> >> we could really do with analyzing all the inlining heuristics with >> every >> DaCapo benchmark and using the optimal set of benchmarks, heuristics, >> etc. From a grep of DaCapo's source, eclipse doesn't show as using >> java.util.Vector - but this could be because it's compressed... >> RVM-446 >> is a nuisance. We'd like to fix the precise type setting of this >> operands for final classes for 2.9.3, but this has broken the compress >> inlining making it too aggressive and regressed its performance. >> This is >> gating the 2.9.3 release. The intended fixes haven't improved >> situation >> and we've also now regressed db. A problem is the age of SpecJVM and >> its >> short execution time - meaning simple inlining decisions can be key to >> making sure we don't cause register allocator difficulties and costing >> us 10s of percent of performance. >> >> Maybe other core team members are aware of/have done work in this >> area? >> It seems like a natural target for machine learning techniques. >> >> Regards, >> Ian >> >> Steve Blackburn wrote: >> >>> Hi Ian, >>> >>> We should give some thought to this. I threw in fop as the profile >>> benchmark after very little consideration. If we're prepared to pay >>> the build-time overhead, I suggest we use a richer benchmark, such as >>> eclipse which is more likely to give the whole VM a shake. I chose >>> fop mainly because it was short running.... What do you (and >>> others) >>> think? >>> >>> I suppose one of us could actually do an evaluation of this and see >>> how each of the options fly... It should not be too hard. I'm >>> prepared to do it, but not at this moment. >>> >>> --Steve >>> >>> On 23/04/2008, at 12:12 PM, captain5050@... wrote: >>> >>> >>> >>>> Revision: 14144 >>>> http://jikesrvm.svn.sourceforge.net/jikesrvm/?rev=14144&view=rev >>>> Author: captain5050 >>>> Date: 2008-04-22 19:12:39 -0700 (Tue, 22 Apr 2008) >>>> >>>> Log Message: >>>> ----------- >>>> Switch from DaCapo fop to lusearch for profiled image runs. This is >>>> an attempt to bump the use of java.util.Vector to increase SpecJVM's >>>> db performance. See RVM-446. >>>> >>>> Modified Paths: >>>> -------------- >>>> rvmroot/trunk/build.xml >>>> >>>> Modified: rvmroot/trunk/build.xml >>>> =================================================================== >>>> --- rvmroot/trunk/build.xml 2008-04-22 15:41:35 UTC (rev 14143) >>>> +++ rvmroot/trunk/build.xml 2008-04-23 02:12:39 UTC (rev 14144) >>>> @@ -1548,7 +1548,7 @@ >>>> <arg value="-X:base:edge_counters=true"/> >>>> <arg value="-X:base:edge_counter_file=${build.profiles}/ >>>> profile.ec"/> >>>> <arg value="-X:aos:final_report_level=2"/> >>>> - <arg line="-jar ${dacapo.jar} -n 3 fop"/> >>>> + <arg line="-jar ${dacapo.jar} -n 3 lusearch"/> >>>> </exec> >>>> </target> >>>> >>>> >>>> >>>> This was sent by the SourceForge.net collaborative development >>>> platform, the world's largest Open Source development site. >>>> >>>> ------------------------------------------------------------------------- >>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>> Don't miss this year's exciting event. There's still time to save >>>> $100. >>>> Use priority code J8TL2D2. >>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>>> _______________________________________________ >>>> Jikesrvm-commits mailing list >>>> Jikesrvm-commits@... >>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-commits >>>> >>>> >>> ------------------------------------------------------------------------- >>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>> Don't miss this year's exciting event. There's still time to save >>> $100. >>> Use priority code J8TL2D2. >>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>> _______________________________________________ >>> Jikesrvm-core mailing list >>> Jikesrvm-core@... >>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >>> >>> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >> Don't miss this year's exciting event. There's still time to save >> $100. >> Use priority code J8TL2D2. >> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >> _______________________________________________ >> Jikesrvm-core mailing list >> Jikesrvm-core@... >> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >> > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > Jikesrvm-core mailing list > Jikesrvm-core@... > https://lists.sourceforge.net/lists/listinfo/jikesrvm-core > ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by Ian Rogers
::
Rate this Message:
David P Grove wrote:
> I'd think (without actually measuring anything....;)) that eclipse > would be > the best choice among the DaCapo benchmarks. > > What we are looking for is something that exercises a fair amount of the VM > codebase in "typical" ways. > Other than the weight of data that eclipse generates being a problem, I guess it is also quite obviously different from most of the Spec benchmarks. A number of the spec benchmarks are fairly direct copies of C code, whereas I believe eclipse is a well engineered piece of OO code. Ian ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
by Steve Blackburn
::
Rate this Message:
On 23/04/2008, at 1:25 PM, Ian Rogers wrote: > Steve Blackburn wrote: >> My feeling is that we have to be very careful not to tune for the >> benchmarks. > > :-) I thought that was the point of benchmarks. Yikes! Benchmarks exist to give us an imperfect indication of how our system might perform in the real world. We want to tune to the real world, not to specific benchmarks. Of course benchmarks are often very useful in identifying performance problems and we should always use them to do that, but we should not be making changes to the VM just so that it will perform on a particular benchmark (unless of course we believe that the performance issue tickled by that benchmark represents an important real world scenario). > Ah, changing the benchmark we profile didn't actually change the > code.. I agree. But as Dave said, it didn't solve the problem either :-) > I think the problem here is that the profiled-image benchmark isn't > giving us good coverage of the libraries and the RVM. For example, fop > has very little exception code in it. Absolutely. It is there more or less by accident. > Maybe the best option is to run > every DaCapo benchmark once. Sure, but that takes a lot of time. > It'd be nice if DaCapo was much less Java 1.4, then when I do > boxing/unboxing improvements I can see a kink in the performance > graphs :-) Sure. It is an open source suite. If you have any suggestions, we're all ears.... ....specially if you're prepared to help out with the non- trivial task of evaluation and packaging (it generally takes weeks to months of full time work to get a benchmark ready---eclipse took me over a month, I think). > Having just sat through many papers at ISPASS on correlating the > similarities between different benchmarks, maybe a similar analysis > can > be applied to DaCapo. See the paper. We did PCA analysis for exactly this reason. > One problem with eclipse is that it covers too > much territory, indeed the most of any of our regular benchmarks. This > can mean we lose any useful profile details in the weight of data we > get. We try to include a mix of workloads in DaCapo. I'm working on a much bigger beast right now. Big complex benchmarks are an important piece of the puzzle. Micro benchmarks also have a very important diagnostic role, but we did not focus on that in DaCapo. --Steve > > > Ian > >> On 23/04/2008, at 12:49 PM, Ian Rogers wrote: >> >> >>> Hi Steve, >>> >>> we could really do with analyzing all the inlining heuristics with >>> every >>> DaCapo benchmark and using the optimal set of benchmarks, >>> heuristics, >>> etc. From a grep of DaCapo's source, eclipse doesn't show as using >>> java.util.Vector - but this could be because it's compressed... >>> RVM-446 >>> is a nuisance. We'd like to fix the precise type setting of this >>> operands for final classes for 2.9.3, but this has broken the >>> compress >>> inlining making it too aggressive and regressed its performance. >>> This is >>> gating the 2.9.3 release. The intended fixes haven't improved >>> situation >>> and we've also now regressed db. A problem is the age of SpecJVM and >>> its >>> short execution time - meaning simple inlining decisions can be >>> key to >>> making sure we don't cause register allocator difficulties and >>> costing >>> us 10s of percent of performance. >>> >>> Maybe other core team members are aware of/have done work in this >>> area? >>> It seems like a natural target for machine learning techniques. >>> >>> Regards, >>> Ian >>> >>> Steve Blackburn wrote: >>> >>>> Hi Ian, >>>> >>>> We should give some thought to this. I threw in fop as the profile >>>> benchmark after very little consideration. If we're prepared to >>>> pay >>>> the build-time overhead, I suggest we use a richer benchmark, >>>> such as >>>> eclipse which is more likely to give the whole VM a shake. I chose >>>> fop mainly because it was short running.... What do you (and >>>> others) >>>> think? >>>> >>>> I suppose one of us could actually do an evaluation of this and see >>>> how each of the options fly... It should not be too hard. I'm >>>> prepared to do it, but not at this moment. >>>> >>>> --Steve >>>> >>>> On 23/04/2008, at 12:12 PM, captain5050@... >>>> wrote: >>>> >>>> >>>> >>>>> Revision: 14144 >>>>> http://jikesrvm.svn.sourceforge.net/jikesrvm/?rev=14144&view=rev >>>>> Author: captain5050 >>>>> Date: 2008-04-22 19:12:39 -0700 (Tue, 22 Apr 2008) >>>>> >>>>> Log Message: >>>>> ----------- >>>>> Switch from DaCapo fop to lusearch for profiled image runs. This >>>>> is >>>>> an attempt to bump the use of java.util.Vector to increase >>>>> SpecJVM's >>>>> db performance. See RVM-446. >>>>> >>>>> Modified Paths: >>>>> -------------- >>>>> rvmroot/trunk/build.xml >>>>> >>>>> Modified: rvmroot/trunk/build.xml >>>>> = >>>>> ================================================================== >>>>> --- rvmroot/trunk/build.xml 2008-04-22 15:41:35 UTC (rev 14143) >>>>> +++ rvmroot/trunk/build.xml 2008-04-23 02:12:39 UTC (rev 14144) >>>>> @@ -1548,7 +1548,7 @@ >>>>> <arg value="-X:base:edge_counters=true"/> >>>>> <arg value="-X:base:edge_counter_file=${build.profiles}/ >>>>> profile.ec"/> >>>>> <arg value="-X:aos:final_report_level=2"/> >>>>> - <arg line="-jar ${dacapo.jar} -n 3 fop"/> >>>>> + <arg line="-jar ${dacapo.jar} -n 3 lusearch"/> >>>>> </exec> >>>>> </target> >>>>> >>>>> >>>>> >>>>> This was sent by the SourceForge.net collaborative development >>>>> platform, the world's largest Open Source development site. >>>>> >>>>> ------------------------------------------------------------------------- >>>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>>> Don't miss this year's exciting event. There's still time to save >>>>> $100. >>>>> Use priority code J8TL2D2. >>>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>>>> _______________________________________________ >>>>> Jikesrvm-commits mailing list >>>>> Jikesrvm-commits@... >>>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-commits >>>>> >>>>> >>>> ------------------------------------------------------------------------- >>>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>>> Don't miss this year's exciting event. There's still time to save >>>> $100. >>>> Use priority code J8TL2D2. >>>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>>> _______________________________________________ >>>> Jikesrvm-core mailing list >>>> Jikesrvm-core@... >>>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >>>> >>>> >>> ------------------------------------------------------------------------- >>> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >>> Don't miss this year's exciting event. There's still time to save >>> $100. >>> Use priority code J8TL2D2. >>> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >>> _______________________________________________ >>> Jikesrvm-core mailing list >>> Jikesrvm-core@... >>> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >>> >> >> >> ------------------------------------------------------------------------- >> This SF.net email is sponsored by the 2008 JavaOne(SM) Conference >> Don't miss this year's exciting event. There's still time to save >> $100. >> Use priority code J8TL2D2. >> http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone >> _______________________________________________ >> Jikesrvm-core mailing list >> Jikesrvm-core@... >> https://lists.sourceforge.net/lists/listinfo/jikesrvm-core >> > > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save > $100. > Use priority code J8TL2D2. > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > Jikesrvm-core mailing list > Jikesrvm-core@... > https://lists.sourceforge.net/lists/listinfo/jikesrvm-core ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone _______________________________________________ Jikesrvm-core mailing list Jikesrvm-core@... https://lists.sourceforge.net/lists/listinfo/jikesrvm-core |
|
|
Re: [rvm-core] [rvm-commits] SF.net SVN: jikesrvm: [14144] rvmroot/trunk/build.xml
|
