|
Benchmark results -- follow-up
Hi,
We are evaluating different options for Open Source ESBs and have come across the WSO2 benchmark results, which puts Mule in a terrible position. However, we haven't been able to find any strong counter-comments from the people who develop Mule.
Therefore, we have a few questions remaining:
1) have the bugs found by the WSO2 team been repaired? How come they hadn't been encountered before running the benchmarks? They are pretty serious problems, it seems.
2) has performance been improved since the results were published?
Anyway, I would appreciate some insight as to why the Mule community has made no move to defend their product, except for the following on the benchmark page itself:
Mule configuration help?
Submitted by ciurana on July 19, 2007 - 06:23.
Hi Asankha!
Interesting results -- thanks for performing the tests.
May I ask how well do you know how to configure Mule? Some of the testing issues that you're having sound like configuration and setup issues, rather than Mule problems. Can you please post the Mule XML configuration file so that we can take a look? Please annotate the file with "objective was X, we did Y, got Z instead" comments. It may be that the configuration you have set up could be optimized.
Thanks!
E
Thanks in advance for your insights.
|
|
Re: Benchmark results -- follow-up
Hi,
Don't forget you are getting your information for a platform that
wants to compete with Mule :) We were never able to run the benchmarks
with the information provided since it's unclear how they ran each
platform. If fact they never originally published the benchmark code,
but I see now it's available since Feb 11th.
The issue they had with keep alive is a configuration error as far as
I can see, and Mule has supported web service proxying for years.
My suggestion is to take the two projects for a test drive if you are
unsure. I think you'll like Mule. Mule has been running in
production environments (including high-performance messaging) for
over 4 years now. It's been battle-tested more that any other open
and closed-source ESB, and provides more functionality out of the box
than the others.
As for defending Mule, it does a very good job of defending itself;
just take it for a test drive.
btw, I'll ask our QA guys to see if there has been enough detail about
this benchmark published so that we could run it. If so we will
happily publish real results for Mule.
Cheers,
Ross Mason
CTO, Co-Founder
MuleSource Inc.
http://mulesource.com | http://blog.rossmason.comOn 9 Apr 2008, at 09:25, raulvk wrote:
>
> Hi,
>
> We are evaluating different options for Open Source ESBs and have come
> across the WSO2 benchmark results, which puts Mule in a terrible
> position.
> However, we haven't been able to find any strong counter-comments
> from the
> people who develop Mule.
>
> Therefore, we have a few questions remaining:
> 1) have the bugs found by the WSO2 team been repaired? How come they
> hadn't been encountered before running the benchmarks? They are pretty
> serious problems, it seems.
> 2) has performance been improved since the results were published?
>
> Anyway, I would appreciate some insight as to why the Mule community
> has
> made no move to defend their product, except for the following on the
> benchmark page itself:
>
>
>
>> Mule configuration help?
>> Submitted by ciurana on July 19, 2007 - 06:23.
>>
>> Hi Asankha!
>>
>> Interesting results -- thanks for performing the tests.
>>
>> May I ask how well do you know how to configure Mule? Some of the
>> testing
>> issues that you're having sound like configuration and setup issues,
>> rather than Mule problems. Can you please post the Mule XML
>> configuration
>> file so that we can take a look? Please annotate the file with
>> "objective
>> was X, we did Y, got Z instead" comments. It may be that the
>> configuration
>> you have set up could be optimized.
>>
>> Thanks!
>>
>> E
>>
>
> Thanks in advance for your insights.
>
> --
> View this message in context: http://www.nabble.com/Benchmark-results----follow-up-tp16590538p16590538.html> Sent from the Mule - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
> http://xircles.codehaus.org/manage_email>
>
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
|
|
Re: Benchmark results -- follow-up
Hi Ross,
Thank you for your answer. I think that the Mule configuration that the WSO2 people used during the benchmark is available here: http://wso2.org/repos/wso2/trunk/commons/performance/esb/mule-1.4.1/.
Could you have someone from your team look at it and let us know your findings? I am very interested in learning your views. To be honest, I think it is important that you do pronounce yourself as soon as possible if the configuration that was used was not optimal, because I'm sure there are many people who've been discouraged to have a further look at Mule due to the inferior results that it achieved on the benchmark (in our case, we were a bit reluctant, since transformation and keepalive connections are imperative in our project).
I'll be looking forward to hearing from you guys. You may contact me at raulvk [ at ] gmail [ dot ] com, if you prefer to speak in private. In the meantime, we will give Mule a test-drive for ourselves.
Thank you very much,
Raul.
Ross Mason-3 wrote:
Hi,
Don't forget you are getting your information for a platform that
wants to compete with Mule :) We were never able to run the benchmarks
with the information provided since it's unclear how they ran each
platform. If fact they never originally published the benchmark code,
but I see now it's available since Feb 11th.
The issue they had with keep alive is a configuration error as far as
I can see, and Mule has supported web service proxying for years.
My suggestion is to take the two projects for a test drive if you are
unsure. I think you'll like Mule. Mule has been running in
production environments (including high-performance messaging) for
over 4 years now. It's been battle-tested more that any other open
and closed-source ESB, and provides more functionality out of the box
than the others.
As for defending Mule, it does a very good job of defending itself;
just take it for a test drive.
btw, I'll ask our QA guys to see if there has been enough detail about
this benchmark published so that we could run it. If so we will
happily publish real results for Mule.
Cheers,
Ross Mason
CTO, Co-Founder
MuleSource Inc.
http://mulesource.com | http://blog.rossmason.comOn 9 Apr 2008, at 09:25, raulvk wrote:
>
> Hi,
>
> We are evaluating different options for Open Source ESBs and have come
> across the WSO2 benchmark results, which puts Mule in a terrible
> position.
> However, we haven't been able to find any strong counter-comments
> from the
> people who develop Mule.
>
> Therefore, we have a few questions remaining:
> 1) have the bugs found by the WSO2 team been repaired? How come they
> hadn't been encountered before running the benchmarks? They are pretty
> serious problems, it seems.
> 2) has performance been improved since the results were published?
>
> Anyway, I would appreciate some insight as to why the Mule community
> has
> made no move to defend their product, except for the following on the
> benchmark page itself:
>
>
>
>> Mule configuration help?
>> Submitted by ciurana on July 19, 2007 - 06:23.
>>
>> Hi Asankha!
>>
>> Interesting results -- thanks for performing the tests.
>>
>> May I ask how well do you know how to configure Mule? Some of the
>> testing
>> issues that you're having sound like configuration and setup issues,
>> rather than Mule problems. Can you please post the Mule XML
>> configuration
>> file so that we can take a look? Please annotate the file with
>> "objective
>> was X, we did Y, got Z instead" comments. It may be that the
>> configuration
>> you have set up could be optimized.
>>
>> Thanks!
>>
>> E
>>
>
> Thanks in advance for your insights.
>
> --
> View this message in context: http://www.nabble.com/Benchmark-results----follow-up-tp16590538p16590538.html> Sent from the Mule - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this list, please visit:
>
> http://xircles.codehaus.org/manage_email>
>
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
|
|
Re: Benchmark results -- follow-up
Raul / Ross
Here are my comments..
Don't forget you are getting your information for a
platform that wants to compete with Mule :) We were never able to run
the benchmarks with the information provided since it's unclear how
they ran each platform. If fact they never originally published the
benchmark code, but I see now it's available since Feb 11th.
I can confirm that we shared necessary information with Eugene
Ciurana on this benchmark after the public comment here..
and offered to re-run the tests with any suggestions from Mule. I
specifically stated " We would be glad to incorporate your
suggestions and to retest and publish new sets of results" ..
However, we never heard back from Eugene or anyone else! Except for
this whining
from Dave :-) (to which Sanjiva
replied)
This comment from
Ruwan confirms that the Mule configuration was available publicly
since 30th of August 2007 (I don't know why you stated Feb 11th date..
?? whats significant about that date??)
The issue they had with keep alive is a configuration
error as far as I can see, and Mule has supported web service proxying
for years.
Well then, why didn't any body from Mule reply to this
email on 8th July 2007?... actually no one has replied to this even
upto now.. I don't think silence in the mailing lists is a good way to
support your many 'production deployments' and 'clients'?..
My suggestion is to take the two projects for a test drive if you are
unsure. I think you'll like Mule. Mule has been running in production
environments (including high-performance messaging) for over 4 years
now. It's been battle-tested more that any other open and
closed-source ESB, and provides more functionality out of the box than
the others.
Let me quote Eugene again, Ref ( http://wso2.org/blog/asankha/2263)
Recently on TSS, Eugene Ciurana was claiming that Mule was able to
do 200 million transactions a day, and that no other ESB they
evaluated, commercial or open source, could do that on the same
hardware. This means that no ESB was able to perform 2315 TPS? I wish
they would describe the scenario tested and the hardware used, instead
of claiming such numbers out loud. For example a single instance of the
WSO2 ESB / Synapse could do well over 3200 TPS with 40 concurrent users
for some scenarios we've tested, on standard hardware (see 'Notes'
here). I am almost sure that Eugene is yet to see the WSO2 ESB /Apache
Synapse in action, cause they are not like the good ol' Mules that
supposedly "Just works"... they work much faster and smarter! and they
are extremely lightweight.
And Eugene replied to this publicly stating "I began looking at your
software on Sunday 15.July, on a recommendation from Joe Ottinger... so
your comment was accurate as of 12.July!" and we haven't heard
from him since then..:)
I am also very interested to see that Mule actually contains
code to recover even from a Java OOM
Exception? I guess this validates your ability to run in production
environments for over 4 years now :-)?
As for defending Mule, it does a very good job of
defending itself; just take it for a test drive.
btw, I'll ask our QA guys to see if there has been enough detail about
this benchmark published so that we could run it. If so we will happily
publish real results for Mule.
Well, lets look at your own comments on Dain
Hansen (BEA)'s blog from November 2006
"Dain, its very misleading to quote your
benchmark figures in a comparison with figures for Mule that were
quoted originally by a user without any context and certainly are not
benchmark figures. We are working on publishing some figures by the
end of the year, but in the meantime maybe you should edit your
posting. Cheers, Ross
Posted by: rossmason on November 10, 2006 at 7:37 AM"
Its been over two (2) years! since this comment Ross.. but why
don't we still see *any* performance benchmarks from Mule yet?
Are you hiding something now ;-) ?
Raul, I agree with Ross on his comment that you should test drive both
Mule and the WSO2 ESB (and/or Apache Synapse). Feel free to contact me
for any help
thanks
asankha
References:
http://wso2.org/library/2259
http://wso2.org/esb
|
|
Re: Benchmark results -- follow-up
Asankha, I am also very interested to see that Mule actually contains
code to recover even from a Java OOM
Exception? I guess this validates your ability to run in production
environments for over 4 years now :-)?
You realize that this kind of statement does not serve the purpose of this discussion. A smiley at the end of it does not make it acceptable either.
You should also realize that what is at stake here is not Mule's credibility but the credibility of a benchmark run by a particular vendor that makes it pass with flying colors. There is a long history of such benchmarks and nothing satisfying never came out of them, at least as far as the end users are concerned.
As an end user, I would rather see you guys, and other ESB vendors, engage a neutral and trusted 3rd party to run such a benchmark. This would help making this discussion less sterile and hopefully more professional.
Thank you,
David
|
|
Re: Benchmark results -- follow-up
Asankha your comments fairly look baised. While we could not test close to a million transactions in a day ( did not have full day available) but We have tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically. Our configurations of course may not be optimal as we were still fine tuning our system. The WSO2 results have been published by they themselves and they are simply another competitor in the same space. To get a true analysis, it would have to be done by a third party nuetral company.
The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
ddossot wrote:
Asankha,
I am also very interested to see that Mule actually contains
code< http://svn.codehaus.org/mule/trunk/mule/transports/file/src/main/java/org/mule/providers/file/transformers/FileToByteArray.java>to
recover even from a Java OOM
> Exception< http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/html/gdaog.html#gbzrr>?
> I guess this validates your ability to run in production environments for
> over 4 years now :-)?
>
You realize that this kind of statement does not serve the purpose of this
discussion. A smiley at the end of it does not make it acceptable either.
You should also realize that what is at stake here is not Mule's credibility
but the credibility of a benchmark run by a particular vendor that makes it
pass with flying colors. There is a long history of such benchmarks and
nothing satisfying never came out of them, at least as far as the end users
are concerned.
As an end user, I would rather see you guys, and other ESB vendors, engage a
neutral and trusted 3rd party to run such a benchmark. This would help
making this discussion less sterile and hopefully more professional.
Thank you,
David
|
|
Re: Benchmark results -- follow-up
David
> You should also realize that what is at stake here is not Mule's
> credibility but the credibility of a benchmark run by a particular
> vendor that makes it pass with flying colors. There is a long history
> of such benchmarks and nothing satisfying never came out of them, at
> least as far as the end users are concerned.
The benchmark we ran, and the configurations used were all made public
and the scenarios explained clearly, and even the tools (such as the
Java ApacheBench clone) shared. We requested anyone to help us fine tune
them, and offered to re-run them. We reported each problem we
encountered during the tests to the Mule team before we published any
results - but we did not get any responses to the issues. You can verify
these statements from http://wso2.org/blog/asankha/2263> As an end user, I would rather see you guys, and other ESB vendors,
> engage a neutral and trusted 3rd party to run such a benchmark. This
> would help making this discussion less sterile and hopefully more
> professional.
See my note about Rosss' own comments on Dain Hansens' blog about Mule
benchmarking itself.. its been over two years since then, but Mule has
never published any performance results, and this only makes one wonder
as to why?
cheers
asankha
---------------------------------------------------------------------
To unsubscribe from this list, please visit:
http://xircles.codehaus.org/manage_email
|
|
Re: Benchmark results -- follow-up
Gary
Asankha your comments fairly look baised.
I must say that I am a lead committer of Apache Synapse and lead the
development of the WSO2 ESB; And I ran the benchmarks against Mule,
ServiceMix and a leading commercial ESB and published the results.
While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.
Interesting.. this translates to just over 6 TPS. Can you explain the
scenarios in more detail?
Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.
To get a true analysis, it would have to be done by a third party nuetral
company.
The reality is you can wait till this "neutral third party" comes up
with an independent benchmark for ESB's, or you can run your own. What
I believe is that as long as you clearly explain the scenarios, make
any code/configuration and tools you use public (and free), and also
ask each vendor for assistance when you find issues and give them an
opportunity to reply, you are good.
The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
Oh no! The first ESB we benchmarked against was a commercial closed
source ESB (See http://wso2.org/library/1721). It was only after we
published those results that we ran the benchmark a second time. And we
included Apache ServiceMix, and ofcourse Mule. We considered OpenESB
and JBoss ESB as well, but due to many issues we faced, we compared
only Apache SM and Mule. You may note that Apache ServiceMix had very
interesting results and did better in transformations initially! And we
did not hide this fact ( http://wso2.org/library/2259), We also reported
the issue we faced against SM and
they looked into it unlike Mule.
asankha
|
|
Re: Benchmark results -- follow-up
Some parts of this message have been removed.
Learn more about Nabble's security policy.
Hi everyone, First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.
I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.
To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.
Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.
Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other. Raul.
On 13/04/2008, Asankha C. Perera <asankha@...> wrote:
Gary
Asankha your comments fairly look baised.
I must say that I am a lead committer of Apache Synapse and lead the
development of the WSO2 ESB; And I ran the benchmarks against Mule,
ServiceMix and a leading commercial ESB and published the results.
While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.
Interesting.. this translates to just over 6 TPS. Can you explain the
scenarios in more detail?
Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.
To get a true analysis, it would have to be done by a third party nuetral
company.
The reality is you can wait till this "neutral third party" comes up
with an independent benchmark for ESB's, or you can run your own. What
I believe is that as long as you clearly explain the scenarios, make
any code/configuration and tools you use public (and free), and also
ask each vendor for assistance when you find issues and give them an
opportunity to reply, you are good.
The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
Oh no! The first ESB we benchmarked against was a commercial closed
source ESB (See http://wso2.org/library/1721). It was only after we
published those results that we ran the benchmark a second time. And we
included Apache ServiceMix, and ofcourse Mule. We considered OpenESB
and JBoss ESB as well, but due to many issues we faced, we compared
only Apache SM and Mule. You may note that Apache ServiceMix had very
interesting results and did better in transformations initially! And we
did not hide this fact ( http://wso2.org/library/2259), We also reported
the issue we faced against SM and
they looked into it unlike Mule.
asankha
|
|
Re: Benchmark results -- follow-up
Hi Mule devs, I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.
If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.
Thanks, Ruwan On Sun, Apr 13, 2008 at 6:15 PM, raulvk < raulvk.soa@...> wrote:
Hi everyone,
First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.
I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.
To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.
Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.
Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.
Raul.
On 13/04/2008, Asankha C. Perera <asankha@...> wrote:
Gary
Asankha your comments fairly look baised.
I must say that I am a lead committer of Apache Synapse and lead the
development of the WSO2 ESB; And I ran the benchmarks against Mule,
ServiceMix and a leading commercial ESB and published the results.
While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.
Interesting.. this translates to just over 6 TPS. Can you explain the
scenarios in more detail?
Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.
To get a true analysis, it would have to be done by a third party nuetral
company.
The reality is you can wait till this "neutral third party" comes up
with an independent benchmark for ESB's, or you can run your own. What
I believe is that as long as you clearly explain the scenarios, make
any code/configuration and tools you use public (and free), and also
ask each vendor for assistance when you find issues and give them an
opportunity to reply, you are good.
The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
Oh no! The first ESB we benchmarked against was a commercial closed
source ESB (See http://wso2.org/library/1721). It was only after we
published those results that we ran the benchmark a second time. And we
included Apache ServiceMix, and ofcourse Mule. We considered OpenESB
and JBoss ESB as well, but due to many issues we faced, we compared
only Apache SM and Mule. You may note that Apache ServiceMix had very
interesting results and did better in transformations initially! And we
did not hide this fact ( http://wso2.org/library/2259), We also reported
the issue we faced against SM and
they looked into it unlike Mule.
asankha
-- Ruwan Linton http://www.wso2.org - "Oxygenating the Web Services Platform"
|
|
Re: Benchmark results -- follow-up
Hi Ruwan,
Thanks for reaching out. We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial. I am surprised there is not an independent body for this yet. I don't think anyone regards users as dumb (not sure why you said this) and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread. Cheers,
Ross Mason CTO, Co-Founder MuleSource Inc.
On 13 Apr 2008, at 18:40, Ruwan Linton wrote: Hi Mule devs,
I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.
If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.
Thanks,
Ruwan
On Sun, Apr 13, 2008 at 6:15 PM, raulvk < raulvk.soa@...> wrote: Hi everyone,
First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.
I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.
To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.
Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.
Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.
Raul. On 13/04/2008, Asankha C. Perera <asankha@...> wrote: Gary
Asankha your comments fairly look baised. I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.
While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically. Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?
Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.
To get a true analysis, it would have to be done by a third party nuetral
company. The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good. The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact ( http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule. asankha
--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"
|
|
Re: Benchmark results -- follow-up
Some parts of this message have been removed.
Learn more about Nabble's security policy.
Hi Ross, Let me speak from the user's point of view. You are very right when you assert that vendors are not typically the best people to run benchmarks. However, I do think that this performance test has not been a *typical* one because usually vendors do not give their competitors a public chance to:
a) correct possible errors/mistakes in the config, prior to publishing the results b) analyse and possibly fix the code used for the benchmarks c) re-run the benchmarks and re-publish a new, fresh set of results
On the other hand, to the WSO2 guys: I would appreciate it if you could provide us with code you used for the implementation of the SimpleStockQuoteService, unless it's the same you've package with the binary distribution of WSO2 ESB, in which case do let us know.
Thank you, Raul. On 14/04/2008, Ross Mason <themuleman@...> wrote:
Hi Ruwan,
Thanks for reaching out. We will run these benchmarks along side our own and publish the results. I do agree with David Dossot here though in that vendors are not the best people to run benchmarks since they are not impartial. I am surprised there is not an independent body for this yet. I don't think anyone regards users as dumb (not sure why you said this) and while users like empirical data they also will not accept benchmark results from a single vendor without doing their own research. That was what I was pushing for in my original response to this thread.
Cheers,
Ross Mason CTO, Co-Founder MuleSource Inc.
On 13 Apr 2008, at 18:40, Ruwan Linton wrote: Hi Mule devs,
I think Raul is exactly correct on this. You should have a look at the configurations (I am sure you already had a deep look in to those :-)) that we used in doing the performance tests and improve the Mule once to optimize the configurations for performance and re-run the performance tests. This will enable the users to understand the performance benchmarks of the two products very easily.
If you need any help in getting Synapse configurations and the product optimized for performance or any information on setting up the environment we would like to incorporate with you. So don't waist any more time and get started with the performance tests. You should bring the counter arguments over Synapse / WSO2 ESB with facts in your hand, otherwise I don't think users will accept those (You should keep in mind that users are not dumb). If you can perform faster than Synapse you don't have to worry at all just show the figures.
Thanks,
Ruwan
On Sun, Apr 13, 2008 at 6:15 PM, raulvk < raulvk.soa@...> wrote:
Hi everyone,
First of all, I'd like to state that I am researching the use of an open source ESB with the uttermost neutrality: our client had determined that they want an open source ESB to serve as their integration backbone, but they have left it up to us to pick the most appropriate one.
I seems that this discussion has deviated and ended up turning into a war between vendors, which doesn't serve us prospective end users at all. My aim when I wrote the first message was to throw some light on why Mule hadn't apparently pronounced themselves on the results.
To the Mule people, even though the benchmark was run by another vendor, the code was released publicly and help was requested on the forums. So there has been a high degree of visibility into the benchmark process. Of course, the WSO2 team are experts in Apache Synapse, which enables them to configure Synapse optimally. That is the only "unfairness" factor that I perceive, but once again, they asked for help on the forums when it came to Mule's config.
Anyway, bottom line is that the only method to address this discussion *productively* is for the Mule team to look at the published benchmark code and point out *where* the configuration mistakes that Ross mentions are, and how they can be fixed. We could then run a new round of benchmarks with an optimised Mule config.
Please consider that the constant goal of both vendors is to gain a larger user base, and *we users* are moved by *facts and figures*, so let's bring them to the table instead of criticising each other.
Raul.
On 13/04/2008, Asankha C. Perera <asankha@...> wrote:
Gary
Asankha your comments fairly look baised. I must say that I am a lead committer of Apache Synapse and lead the development of the WSO2 ESB; And I ran the benchmarks against Mule, ServiceMix and a leading commercial ESB and published the results.
While we could not test close to a million transactions in a day ( did not have full day available) but We have
tested Mule with close to a 100000 transactions in 4-5 hours and it performed fanstatically.
Interesting.. this translates to just over 6 TPS. Can you explain the scenarios in more detail?
Our configurations of course may not be optimal as
we were still fine tuning our system. The WSO2 results have been published
by they themselves and they are simply another competitor in the same space.
To get a true analysis, it would have to be done by a third party nuetral
company. The reality is you can wait till this "neutral third party" comes up with an independent benchmark for ESB's, or you can run your own. What I believe is that as long as you clearly explain the scenarios, make any code/configuration and tools you use public (and free), and also ask each vendor for assistance when you find issues and give them an opportunity to reply, you are good.
The fact that WSO2 is basically trying to bench mark against mule suggests in iteslf that WSO2 acknowledges that Mule may be "The Standard" to beat. Better to go with the "The Standard" I guess.
Oh no! The first ESB we benchmarked against was a commercial closed source ESB (See http://wso2.org/library/1721). It was only after we published those results that we ran the benchmark a second time. And we included Apache ServiceMix, and ofcourse Mule. We considered OpenESB and JBoss ESB as well, but due to many issues we faced, we compared only Apache SM and Mule. You may note that Apache ServiceMix had very interesting results and did better in transformations initially! And we did not hide this fact ( http://wso2.org/library/2259), We also reported the issue we faced against SM and they looked into it unlike Mule.
asankha
--
Ruwan Linton
http://www.wso2.org - "Oxygenating the Web Services Platform"
|
|
Re: Benchmark results -- follow-up
Hi Ross, Nice to see that you are going to run the performance tests. |
