|
View:
New views
14 Messages
—
Rating Filter:
Alert me
|
|
|
Web server and tomcat - firewallHi all|
Maybe you could help me to solve a problem with your experience. My company has implemented a solution of 3 layers (Apache web server, Tomcat Container and Database). There is a firewall between the web server and the tomcat. We get the following errors in mod_jk log: These errors are caused by loss of TCP packages in firewall (???). [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c (951): (p11) can't receive the response message from tomcat, network problems or tomcat ( 10.202.131.130:8009) is down (errno=54) [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c (1592): (p11) Tomcat is down or refused connection. No response has been sent to the client (yet) (is not true when log says that tomcat is down because some requests get the correct response) Setting "JkOptions + DisableReuse" in httpd.conf does not solve the problem. Any suggestions? Thank you and I apologise for the inconvenience. Greetings |
|
|
Re: Web server and tomcat - firewallcould be the extra space JkOptions + DisableReuse should be JkOptions +DisableReuse someone will correct me if I am wrong :) if you are not using disable reuse, one has to match up the max clients on the httpd side with the maxThreads on the tomcat side, so that you don't leave connections hanging. do a "netstat -na |grep <ajp port number>" on both the httpd machine and the tomcat machine, to see the state of the AJP connections Filip Andrew Hole wrote: > Hi all| > > > > Maybe you could help me to solve a problem with your experience. > > My company has implemented a solution of 3 layers (Apache web server, Tomcat > Container and Database). There is a firewall between the web server and the > tomcat. > > We get the following errors in mod_jk log: > > > > These errors are caused by loss of TCP packages in firewall (???). > > [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c (951): (p11) > can't receive the response message from tomcat, network problems or tomcat ( > 10.202.131.130:8009) is down (errno=54) > > [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c (1592): (p11) > Tomcat is down or refused connection. No response has been sent to the > client (yet) > > (is not true when log says that tomcat is down because some requests get the > correct response) > > > > Setting "JkOptions + DisableReuse" in httpd.conf does not solve the problem. > Any suggestions? > > > > Thank you and I apologise for the inconvenience. > > > > Greetings > > > ------------------------------------------------------------------------ > > No virus found in this incoming message. > Checked by AVG. > Version: 7.5.524 / Virus Database: 269.23.9/1417 - Release Date: 5/6/2008 8:07 AM > --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Web server and tomcat - firewallHi Andrew,
Andrew Hole schrieb: > Hi all| > > > > Maybe you could help me to solve a problem with your experience. > > My company has implemented a solution of 3 layers (Apache web server, Tomcat > Container and Database). There is a firewall between the web server and the > tomcat. > > We get the following errors in mod_jk log: > > > > These errors are caused by loss of TCP packages in firewall (???). How do you know? > [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c (951): (p11) > can't receive the response message from tomcat, network problems or tomcat ( > 10.202.131.130:8009) is down (errno=54) What is errno 54 on your platform? perl -e 'use POSIX;print POSIX::strerror(54),"\n"' > [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c (1592): (p11) > Tomcat is down or refused connection. No response has been sent to the > client (yet) > > (is not true when log says that tomcat is down because some requests get the > correct response) > > > > Setting "JkOptions + DisableReuse" in httpd.conf does not solve the problem. > Any suggestions? DisableReuse will have a huge performance hit, but will eliminate any problems w.r.t. reusing idle connections, which might have been dropped by the firewall. Post version of mod_jk and configuration. > Thank you and I apologise for the inconvenience. > > > > Greetings Regards, Rainer --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Web server and tomcat - firewall-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1 Rainer, Rainer Jung wrote: |> [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c |> (951): (p11) can't receive the response message from tomcat, |> network problems or tomcat ( 10.202.131.130:8009) is down |> (errno=54) | | What is errno 54 on your platform? FWIW, errno 54 on my Linux 2.6/glibc 2.6.1 is: "Error 54: Exchange full" ??! - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkghsGUACgkQ9CaO5/Lv0PDB0ACdEkQf0AfaJAGV+4lEYhuvhMlo N/0AmwfQkoxH5H4ypV/GqSFEgP3dZw43 =wj3O -----END PGP SIGNATURE----- --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Web server and tomcat - firewallFilip Hanik - Dev Lists wrote:
> > could be the extra space > > JkOptions + DisableReuse > > should be > > JkOptions +DisableReuse > > someone will correct me if I am wrong :) Oupss, didn't take the space serious. Of course it should not be there. Although if we don't recognize the option, we should throw out a "JkOptions: Illegal option" message (don't remember since what version though). --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Web server and tomcat - firewallChristopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Rainer, > > Rainer Jung wrote: > |> [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c > |> (951): (p11) can't receive the response message from tomcat, > |> network problems or tomcat ( 10.202.131.130:8009) is down > |> (errno=54) > | > | What is errno 54 on your platform? > > FWIW, errno 54 on my Linux 2.6/glibc 2.6.1 is: > > "Error 54: Exchange full" On the Linux box I checked it's the same, but the OP didn't give any indication about the platform, and the error text was so unusual, that i suspect he's using BSD or Mac OS or whatever might have a very different meaning for 54. Unfortunately most errnos are not standardized. > ??! Regards, Rainer --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Web server and tomcat - firewallHI!
I don't have perl installed in production servers, but it is windows server 2003. About mod_jk configuration (mod_jk/1.2.18) : # List the workers name worker.list = p1balancer, resumop1 # ---------------------- # Load Balancer worker # ---------------------- worker.p1balancer.type= lb worker.p1balancer.balanced_workers= p11,p12 # ---------------- # First worker # ---------------- worker.p11.port= 8009 worker.p11.host= 10.202.131.130 worker.p11.type= ajp13 # ---------------- # Second worker # ---------------- worker.p12.port= 8009 worker.p12.host= 10.202.131.131 worker.p12.type= ajp13 # ---------------------- # Load Balancer worker # ---------------------- worker.resumop1.type= lb worker.resumop1.balanced_workers= rp11,rp12 # ---------------- # First worker # ---------------- worker.rp11.port= 8007 worker.rp11.host= 10.202.131.130 worker.rp11.type= ajp13 # ---------------- # Second worker # ---------------- worker.rp12.port= 8007 worker.rp12.host= 10.202.131.131 worker.rp12.type= ajp13 On tomcat side <Connector port="*8009*" enableLookups="*false*"redirectPort ="*8443*" protocol="*AJP/1.3*" /> Thanks a lot On Wed, May 7, 2008 at 2:36 PM, Christopher Schultz < chris@...> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Rainer, > > Rainer Jung wrote: > |> [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c > |> (951): (p11) can't receive the response message from tomcat, > |> network problems or tomcat ( 10.202.131.130:8009) is down > |> (errno=54) > | > | What is errno 54 on your platform? > > FWIW, errno 54 on my Linux 2.6/glibc 2.6.1 is: > > "Error 54: Exchange full" > > ??! > > - -chris > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.9 (MingW32) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iEYEARECAAYFAkghsGUACgkQ9CaO5/Lv0PDB0ACdEkQf0AfaJAGV+4lEYhuvhMlo > N/0AmwfQkoxH5H4ypV/GqSFEgP3dZw43 > =wj3O > -----END PGP SIGNATURE----- > > > --------------------------------------------------------------------- > To start a new topic, e-mail: users@... > To unsubscribe, e-mail: users-unsubscribe@... > For additional commands, e-mail: users-help@... > > |
|
|
Re: Web server and tomcat - firewall-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1 Andrew, Andrew Hole wrote: | I don't have perl installed in production servers, but it is windows server | 2003. You could always try this: #include <stdio.h> #include <string.h> #include <stdlib.h> #define MAX_ERROR_STRING_LENGTH 1024 int main(int argc, char *argv[]) { ~ int error_num; ~ char error_msg[MAX_ERROR_STRING_LENGTH]; ~ if(argc < 2) { ~ printf("Usage: %s errno\n", argv[0]); ~ return 1; ~ } ~ error_num = atoi(argv[1]); ~ strerror_r(error_num, error_msg, MAX_ERROR_STRING_LENGTH); ~ printf("Error %d: %s\n", error_num, error_msg); ~ return 0; } Also, if you are using MySQL, it comes with a "perror" program that basically does what the above does (but also reports MySQL error numbers, too). Not sure if that's included in the win32 build, though. - -chris -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkghzrwACgkQ9CaO5/Lv0PAA0gCePllyQnGBc1AGsOJuzf+gk/cH 6bkAn3xqSAHYeIJq006pR1wskBNMUG8i =QaJQ -----END PGP SIGNATURE----- --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Web server and tomcat - firewallAndrew Hole wrote:
> HI! > > I don't have perl installed in production servers, but it is windows server > 2003. OK, then it is windows error 10054 = Connection reset by peer. > About mod_jk configuration (mod_jk/1.2.18) : 1.2.18 does not have DisableReuse. It was first implemented in 1.2.19. Bad luck. If you upgrade (you should), then go directly to 1.2.26. > # List the workers name > worker.list = p1balancer, resumop1 > > # ---------------------- > # Load Balancer worker > # ---------------------- > worker.p1balancer.type= lb > worker.p1balancer.balanced_workers= p11,p12 > > # ---------------- > # First worker > # ---------------- > worker.p11.port= 8009 > worker.p11.host= 10.202.131.130 > worker.p11.type= ajp13 > > > # ---------------- > # Second worker > # ---------------- > worker.p12.port= 8009 > worker.p12.host= 10.202.131.131 > worker.p12.type= ajp13 > > > > # ---------------------- > # Load Balancer worker > # ---------------------- > worker.resumop1.type= lb > worker.resumop1.balanced_workers= rp11,rp12 > > # ---------------- > # First worker > # ---------------- > worker.rp11.port= 8007 > worker.rp11.host= 10.202.131.130 > worker.rp11.type= ajp13 > > > # ---------------- > # Second worker > # ---------------- > worker.rp12.port= 8007 > worker.rp12.host= 10.202.131.131 > worker.rp12.type= ajp13 > > > On tomcat side <Connector port="*8009*" enableLookups="*false*"redirectPort > ="*8443*" protocol="*AJP/1.3*" /> > > Thanks a lot You should start with reading the (relatively) new docs page about timeouts: http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html They will help detect such a problem and recover more gracefully. Especially cping/cpong are helpful in early problem detection. The pool timeouts should be helpful for closing idle connections, before you get a connection drop. Try to understand the page, and if things are unclear, ask us. How do you know, that the firewall is the root cause? To get a better understanding for the situation, you could compare "netstat -an" on the apache httpd side with one on the tomcat side. Look only for the relevant ports (8007 resp. 8009) and look for differences between the output. There could be small differences for a short period of time, but there should be no connections missing or being in a different state between the two outputs for a longer time (except for TIME_WAIT). Regards, Rainer > On Wed, May 7, 2008 at 2:36 PM, Christopher Schultz < > chris@...> wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Rainer, >> >> Rainer Jung wrote: >> |> [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c >> |> (951): (p11) can't receive the response message from tomcat, >> |> network problems or tomcat ( 10.202.131.130:8009) is down >> |> (errno=54) >> | >> | What is errno 54 on your platform? >> >> FWIW, errno 54 on my Linux 2.6/glibc 2.6.1 is: >> >> "Error 54: Exchange full" >> >> ??! >> >> - -chris --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
503 under loadHi,
We've started to stress test a new website, and are seeing 503's interspersed when we are under load. We are running Apache - Tomcat combination. Could someone give me an idea of what can cause this, and where to start looking? cheers, David |
|
|
Re: 503 under loaddhay@... wrote:
> Hi, > > We've started to stress test a new website, and are seeing 503's > interspersed when we are under load. We are running Apache - Tomcat > combination. > > Could someone give me an idea of what can cause this, and where to start > looking? > > cheers, > > David > > 1)What is the maxclients settings in httpd.conf (Apache) 2)Any specific errors in catalina logs (Tomcat) 3)Do you have ample RAM and CPU for the test (please elaborate on hardware) 4)Database overload? (elaborate on DB if any) -- Regards Gabe Wong NGASI AppServer Manager JAVA AUTOMATION and SaaS Enablement http://www.ngasi.com NEW! 8.1 - Centrally manage Application Servers across multiple physical servers --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: 503 under loadHi David,
dhay@... schrieb: > Hi, > > We've started to stress test a new website, and are seeing 503's > interspersed when we are under load. We are running Apache - Tomcat > combination. > > Could someone give me an idea of what can cause this, and where to start > looking? I expect the 503 is generated by httpd/mod_jk. If so, then your mod_jk log file should give an idea, if httpd couldn't contact Tomcat, didn't get an answer, ... Is your stress test using a defined maximum parallelity? If not and your test client uses a timeout, then once your app gets slow, you'll easily saturate all Tomcat threads, DB connections, ... and finally httpd will no longer be able to send new requests to Tomcat. Du regular thread dumps to your Tomcat while being under stress, to check what the AJP threads (TP-Processor) are actually doing. > > cheers, > > David Regards, Rainer --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
|
|
Re: Web server and tomcat - firewallThanks
What means: error 10054 = Connection reset by peer. Andrew On Wed, May 7, 2008 at 7:14 PM, Rainer Jung <rainer.jung@...> wrote: > Andrew Hole wrote: > > > HI! > > > > I don't have perl installed in production servers, but it is windows > > server > > 2003. > > > > OK, then it is windows error 10054 = Connection reset by peer. > > About mod_jk configuration (mod_jk/1.2.18) : > > > > 1.2.18 does not have DisableReuse. It was first implemented in 1.2.19. Bad > luck. If you upgrade (you should), then go directly to 1.2.26. > > > # List the workers name > > worker.list = p1balancer, resumop1 > > > > # ---------------------- > > # Load Balancer worker > > # ---------------------- > > worker.p1balancer.type= lb > > worker.p1balancer.balanced_workers= p11,p12 > > > > # ---------------- > > # First worker > > # ---------------- > > worker.p11.port= 8009 > > worker.p11.host= 10.202.131.130 > > worker.p11.type= ajp13 > > > > > > # ---------------- > > # Second worker > > # ---------------- > > worker.p12.port= 8009 > > worker.p12.host= 10.202.131.131 > > worker.p12.type= ajp13 > > > > > > > > # ---------------------- > > # Load Balancer worker > > # ---------------------- > > worker.resumop1.type= lb > > worker.resumop1.balanced_workers= rp11,rp12 > > > > # ---------------- > > # First worker > > # ---------------- > > worker.rp11.port= 8007 > > worker.rp11.host= 10.202.131.130 > > worker.rp11.type= ajp13 > > > > > > # ---------------- > > # Second worker > > # ---------------- > > worker.rp12.port= 8007 > > worker.rp12.host= 10.202.131.131 > > worker.rp12.type= ajp13 > > > > > > On tomcat side <Connector port="*8009*" > > enableLookups="*false*"redirectPort > > ="*8443*" protocol="*AJP/1.3*" /> > > > > Thanks a lot > > > > You should start with reading the (relatively) new docs page about > timeouts: > > http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html > > They will help detect such a problem and recover more gracefully. > Especially cping/cpong are helpful in early problem detection. > > The pool timeouts should be helpful for closing idle connections, before > you get a connection drop. Try to understand the page, and if things are > unclear, ask us. > > How do you know, that the firewall is the root cause? > > To get a better understanding for the situation, you could compare > "netstat -an" on the apache httpd side with one on the tomcat side. Look > only for the relevant ports (8007 resp. 8009) and look for differences > between the output. There could be small differences for a short period of > time, but there should be no connections missing or being in a different > state between the two outputs for a longer time (except for TIME_WAIT). > > Regards, > > Rainer > > On Wed, May 7, 2008 at 2:36 PM, Christopher Schultz < > > chris@...> wrote: > > > > -----BEGIN PGP SIGNED MESSAGE----- > > > Hash: SHA1 > > > > > > Rainer, > > > > > > Rainer Jung wrote: > > > |> [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c > > > |> (951): (p11) can't receive the response message from tomcat, > > > |> network problems or tomcat ( 10.202.131.130:8009) is down > > > |> (errno=54) > > > | > > > | What is errno 54 on your platform? > > > > > > FWIW, errno 54 on my Linux 2.6/glibc 2.6.1 is: > > > > > > "Error 54: Exchange full" > > > > > > ??! > > > > > > - -chris > > > > > > --------------------------------------------------------------------- > To start a new topic, e-mail: users@... > To unsubscribe, e-mail: users-unsubscribe@... > For additional commands, e-mail: users-help@... > > |
|
|
Re: Web server and tomcat - firewallAndrew Hole schrieb:
> Thanks > > What means: error 10054 = Connection reset by peer. The backend (Tomcat) or the firewall or something similar in between httpd and Tomcat send back a TCP reset packet, when mod_jk send a packet on an established connection. A reset connection can not be used any longer. Rainer > Andrew > > On Wed, May 7, 2008 at 7:14 PM, Rainer Jung <rainer.jung@...> wrote: > >> Andrew Hole wrote: >> >>> HI! >>> >>> I don't have perl installed in production servers, but it is windows >>> server >>> 2003. >>> >> OK, then it is windows error 10054 = Connection reset by peer. >> >> About mod_jk configuration (mod_jk/1.2.18) : >> 1.2.18 does not have DisableReuse. It was first implemented in 1.2.19. Bad >> luck. If you upgrade (you should), then go directly to 1.2.26. >> >> >> # List the workers name >>> worker.list = p1balancer, resumop1 >>> >>> # ---------------------- >>> # Load Balancer worker >>> # ---------------------- >>> worker.p1balancer.type= lb >>> worker.p1balancer.balanced_workers= p11,p12 >>> >>> # ---------------- >>> # First worker >>> # ---------------- >>> worker.p11.port= 8009 >>> worker.p11.host= 10.202.131.130 >>> worker.p11.type= ajp13 >>> >>> >>> # ---------------- >>> # Second worker >>> # ---------------- >>> worker.p12.port= 8009 >>> worker.p12.host= 10.202.131.131 >>> worker.p12.type= ajp13 >>> >>> >>> >>> # ---------------------- >>> # Load Balancer worker >>> # ---------------------- >>> worker.resumop1.type= lb >>> worker.resumop1.balanced_workers= rp11,rp12 >>> >>> # ---------------- >>> # First worker >>> # ---------------- >>> worker.rp11.port= 8007 >>> worker.rp11.host= 10.202.131.130 >>> worker.rp11.type= ajp13 >>> >>> >>> # ---------------- >>> # Second worker >>> # ---------------- >>> worker.rp12.port= 8007 >>> worker.rp12.host= 10.202.131.131 >>> worker.rp12.type= ajp13 >>> >>> >>> On tomcat side <Connector port="*8009*" >>> enableLookups="*false*"redirectPort >>> ="*8443*" protocol="*AJP/1.3*" /> >>> >>> Thanks a lot >>> >> You should start with reading the (relatively) new docs page about >> timeouts: >> >> http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html >> >> They will help detect such a problem and recover more gracefully. >> Especially cping/cpong are helpful in early problem detection. >> >> The pool timeouts should be helpful for closing idle connections, before >> you get a connection drop. Try to understand the page, and if things are >> unclear, ask us. >> >> How do you know, that the firewall is the root cause? >> >> To get a better understanding for the situation, you could compare >> "netstat -an" on the apache httpd side with one on the tomcat side. Look >> only for the relevant ports (8007 resp. 8009) and look for differences >> between the output. There could be small differences for a short period of >> time, but there should be no connections missing or being in a different >> state between the two outputs for a longer time (except for TIME_WAIT). >> >> Regards, >> >> Rainer >> >> On Wed, May 7, 2008 at 2:36 PM, Christopher Schultz < >>> chris@...> wrote: >>> >>> -----BEGIN PGP SIGNED MESSAGE----- >>>> Hash: SHA1 >>>> >>>> Rainer, >>>> >>>> Rainer Jung wrote: >>>> |> [Tue May 06 16:24:03 2008] [2704:3128] [error] jk_ajp_common.c >>>> |> (951): (p11) can't receive the response message from tomcat, >>>> |> network problems or tomcat ( 10.202.131.130:8009) is down >>>> |> (errno=54) >>>> | >>>> | What is errno 54 on your platform? >>>> >>>> FWIW, errno 54 on my Linux 2.6/glibc 2.6.1 is: >>>> >>>> "Error 54: Exchange full" >>>> >>>> ??! >>>> >>>> - -chris >>>> >> --------------------------------------------------------------------- >> To start a new topic, e-mail: users@... >> To unsubscribe, e-mail: users-unsubscribe@... >> For additional commands, e-mail: users-help@... >> >> > --------------------------------------------------------------------- To start a new topic, e-mail: users@... To unsubscribe, e-mail: users-unsubscribe@... For additional commands, e-mail: users-help@... |
| Free Forum Powered by Nabble | Forum Help |