Apache/mod_jk serves random files from tomcat

[Tim Redding]

Hi,

We are experiencing intermittent problems with a particular site that is not returning the correct file that is requested.  For instance if we request the index.html file we actually get a css file or even an image.  From the apache access log you can see that the size of the index.html file grows on the second request. This is because a gif was actually returned.

XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] "GET /index.html HTTP/1.1" 200 1068
XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] "GET /index.html HTTP/1.1" 200 9526
XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] "GET /index.html HTTP/1.1" 200 1086

No error messages are logged in the mode_jk.log file.

We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk (version unknown but fairly recent).  We have all assets in our war file.  When we hit Tomcat directly on port 8080 it serves the correct file. And to fix the problem an apache restart seems to sort things out.

On this server with have 2 vhosts.  One is a simple nothing fancy static site and the other forwards everything to our Tomcat server.  Below I've included our mod_jk config and a snippet of our httpd.conf.  

Any ideas or things to try would be most appreciated.


Tim.



============= mod_jk.conf ==========

# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so

# Where to find workers.properties
JkWorkersFile conf/workers.properties

# Where to put jk logs
JkLogFile logs/mod_jk.log

# Set the jk log level [debug/error/info]
JkLogLevel debug

# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"

# JkOptions indicates to send SSK KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories

# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"

# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm

# original URL pass through
JkEnvVar    ORIGINAL_URI    w00t

# Add jkstatus for managing runtime data
<Location /jkstatus/>
JkMount status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>


======= httpd.conf (our additions to the default file) ======

# mod_jk include
Include conf/mod_jk.conf

<VirtualHost *:80>
    DocumentRoot /var/www/html/
    ServerName example.co.uk
    ErrorLog logs/default-error.log
    CustomLog logs/default-access.log common
    alias /logs /var/widgets
    <Location /logs>
        AuthUserFile /var/widgets/.htpasswd
        AuthName "Widgets"
        AuthType Basic
        Require valid-user
    </Location>
 
    Rewriteengine on
    RewriteRule ^/$ /index.html [R]
    jkmount /* loadbalancer
    jkunmount /logs/*.gz loadbalancer
</VirtualHost>

<VirtualHost *:80>
    DocumentRoot /var/www/html/
    ServerName widgets.example.co.uk
    ErrorLog /var/widgets/widget-error.log
    CustomLog /var/widgets/widgets-access.log common
    jkunmount /* loadbalancer
</VirtualHost>


======= worker.properties ======

worker.list=loadbalancer,status
worker.node1.port=8009
worker.node1.host=127.0.0.1
worker.node1.type=ajp13
worker.node1.lbfactor=1
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=node1
worker.status.type=status

[Len Popp]

That log file is from the httpd server, right? What does the Tomcat
log file say? (Turn on AccessLogValve if you haven't already.) Is
Tomcat always getting requests for the correct file, or is mod_jk
requesting the wrong file sometimes?
--
Len


On Thu, Jul 10, 2008 at 11:44, Tim Redding <timr@...> wrote:

>
> Hi,
>
> We are experiencing intermittent problems with a particular site that is not
> returning the correct file that is requested.  For instance if we request
> the index.html file we actually get a css file or even an image.  From the
> apache access log you can see that the size of the index.html file grows on
> the second request. This is because a gif was actually returned.
>
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] "GET /index.html HTTP/1.1"
> 200 1068
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] "GET /index.html HTTP/1.1"
> 200 9526
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] "GET /index.html HTTP/1.1"
> 200 1086
>
> No error messages are logged in the mode_jk.log file.
>
> We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk
> (version unknown but fairly recent).  We have all assets in our war file.
> When we hit Tomcat directly on port 8080 it serves the correct file. And to
> fix the problem an apache restart seems to sort things out.
>
> On this server with have 2 vhosts.  One is a simple nothing fancy static
> site and the other forwards everything to our Tomcat server.  Below I've
> included our mod_jk config and a snippet of our httpd.conf.
>
> Any ideas or things to try would be most appreciated.
>
>
> Tim.
>
>
>
> ============= mod_jk.conf ==========
>
> # Load mod_jk module
> # Specify the filename of the mod_jk lib
> LoadModule jk_module modules/mod_jk.so
>
> # Where to find workers.properties
> JkWorkersFile conf/workers.properties
>
> # Where to put jk logs
> JkLogFile logs/mod_jk.log
>
> # Set the jk log level [debug/error/info]
> JkLogLevel debug
>
> # Select the log format
> JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"
>
> # JkOptions indicates to send SSK KEY SIZE
> JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
>
> # JkRequestLogFormat
> JkRequestLogFormat "%w %V %T"
>
> # Add shared memory.
> # This directive is present with 1.2.10 and
> # later versions of mod_jk, and is needed for
> # for load balancing to work properly
> JkShmFile logs/jk.shm
>
> # original URL pass through
> JkEnvVar    ORIGINAL_URI    w00t
>
> # Add jkstatus for managing runtime data
> <Location /jkstatus/>
> JkMount status
> Order deny,allow
> Deny from all
> Allow from 127.0.0.1
> </Location>
>
>
> ======= httpd.conf (our additions to the default file) ======
>
> # mod_jk include
> Include conf/mod_jk.conf
>
> <VirtualHost *:80>
>    DocumentRoot /var/www/html/
>    ServerName example.co.uk
>    ErrorLog logs/default-error.log
>    CustomLog logs/default-access.log common
>    alias /logs /var/widgets
>    <Location /logs>
>        AuthUserFile /var/widgets/.htpasswd
>        AuthName "Widgets"
>        AuthType Basic
>        Require valid-user
>    </Location>
>
>    Rewriteengine on
>    RewriteRule ^/$ /index.html [R]
>    jkmount /* loadbalancer
>    jkunmount /logs/*.gz loadbalancer
> </VirtualHost>
>
> <VirtualHost *:80>
>    DocumentRoot /var/www/html/
>    ServerName widgets.example.co.uk
>    ErrorLog /var/widgets/widget-error.log
>    CustomLog /var/widgets/widgets-access.log common
>    jkunmount /* loadbalancer
> </VirtualHost>
>
>
> ======= worker.properties ======
>
> worker.list=loadbalancer,status
> worker.node1.port=8009
> worker.node1.host=127.0.0.1
> worker.node1.type=ajp13
> worker.node1.lbfactor=1
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=node1
> worker.status.type=status
> --
> View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html
> Sent from the Tomcat - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim,

Tim Redding wrote:
| We are experiencing intermittent problems with a particular site that
is not
| returning the correct file that is requested.  For instance if we request
| the index.html file we actually get a css file or even an image.  From the
| apache access log you can see that the size of the index.html file
grows on
| the second request. This is because a gif was actually returned.
|
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] "GET /index.html
HTTP/1.1"
| 200 1068
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] "GET /index.html
HTTP/1.1"
| 200 9526
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] "GET /index.html
HTTP/1.1"
| 200 1086

Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?

Does this happen whether Tomcat is running or not? How about if mod_jk
is disabled versus enabled?

| We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk
| (version unknown but fairly recent).

$ strings /path/to/mod_jk.so | grep mod_jk/
mod_jk/1.2.26

(This is what mine returns). Remember that mod_jk 1.2 has been around
for ... ever and there's a big difference between 1.2.5 and 1.2.26.
Virtually all versions of mod_jk are compatible with all versions of
Apache httpd and Tomcat, so you should upgrade to the latest whenever it
is convenient for you.

| We have all assets in our war file.

Does the WAR file get expanded during deployment?

| JkMount status

Probably not your problem, but is this the correct syntax? I would have
expected something like

JkMount status_uri status

|     jkmount /* loadbalancer

Looks like everything goes to Tomcat. Try enabling the AccessLogValve to
see which requests Tomcat is responding to -- that may help shed a bit
of light on the problem.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkh2eEsACgkQ9CaO5/Lv0PDSZwCgjk7OvOHHAZpvDDolD3JAgIdq
EVgAnRWBvsQbbNZlSvJsRp+b2dLmT0ml
=9ZiZ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Tim Redding]

Thanks for the quick reply.  I've enabled the AccessLogValve.  I've just gotta wait for it to start playing up again.  Could be 2 hours or 2 weeks.  I'll reply when I have more info.

Tim.

Len Popp wrote:

That log file is from the httpd server, right? What does the Tomcat
log file say? (Turn on AccessLogValve if you haven't already.) Is
Tomcat always getting requests for the correct file, or is mod_jk
requesting the wrong file sometimes?
--
Len


On Thu, Jul 10, 2008 at 11:44, Tim Redding <[e-mail]> wrote:
>
> Hi,
>
> We are experiencing intermittent problems with a particular site that is not
> returning the correct file that is requested.  For instance if we request
> the index.html file we actually get a css file or even an image.  From the
> apache access log you can see that the size of the index.html file grows on
> the second request. This is because a gif was actually returned.
>
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] "GET /index.html HTTP/1.1"
> 200 1068
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] "GET /index.html HTTP/1.1"
> 200 9526
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] "GET /index.html HTTP/1.1"
> 200 1086
>
> No error messages are logged in the mode_jk.log file.
>
> We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk
> (version unknown but fairly recent).  We have all assets in our war file.
> When we hit Tomcat directly on port 8080 it serves the correct file. And to
> fix the problem an apache restart seems to sort things out.
>
> On this server with have 2 vhosts.  One is a simple nothing fancy static
> site and the other forwards everything to our Tomcat server.  Below I've
> included our mod_jk config and a snippet of our httpd.conf.
>
> Any ideas or things to try would be most appreciated.
>
>
> Tim.
>
>
>
> ============= mod_jk.conf ==========
>
> # Load mod_jk module
> # Specify the filename of the mod_jk lib
> LoadModule jk_module modules/mod_jk.so
>
> # Where to find workers.properties
> JkWorkersFile conf/workers.properties
>
> # Where to put jk logs
> JkLogFile logs/mod_jk.log
>
> # Set the jk log level [debug/error/info]
> JkLogLevel debug
>
> # Select the log format
> JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"
>
> # JkOptions indicates to send SSK KEY SIZE
> JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
>
> # JkRequestLogFormat
> JkRequestLogFormat "%w %V %T"
>
> # Add shared memory.
> # This directive is present with 1.2.10 and
> # later versions of mod_jk, and is needed for
> # for load balancing to work properly
> JkShmFile logs/jk.shm
>
> # original URL pass through
> JkEnvVar    ORIGINAL_URI    w00t
>
> # Add jkstatus for managing runtime data
> <Location /jkstatus/>
> JkMount status
> Order deny,allow
> Deny from all
> Allow from 127.0.0.1
> </Location>
>
>
> ======= httpd.conf (our additions to the default file) ======
>
> # mod_jk include
> Include conf/mod_jk.conf
>
> <VirtualHost *:80>
>    DocumentRoot /var/www/html/
>    ServerName example.co.uk
>    ErrorLog logs/default-error.log
>    CustomLog logs/default-access.log common
>    alias /logs /var/widgets
>    <Location /logs>
>        AuthUserFile /var/widgets/.htpasswd
>        AuthName "Widgets"
>        AuthType Basic
>        Require valid-user
>    </Location>
>
>    Rewriteengine on
>    RewriteRule ^/$ /index.html [R]
>    jkmount /* loadbalancer
>    jkunmount /logs/*.gz loadbalancer
> </VirtualHost>
>
> <VirtualHost *:80>
>    DocumentRoot /var/www/html/
>    ServerName widgets.example.co.uk
>    ErrorLog /var/widgets/widget-error.log
>    CustomLog /var/widgets/widgets-access.log common
>    jkunmount /* loadbalancer
> </VirtualHost>
>
>
> ======= worker.properties ======
>
> worker.list=loadbalancer,status
> worker.node1.port=8009
> worker.node1.host=127.0.0.1
> worker.node1.type=ajp13
> worker.node1.lbfactor=1
> worker.loadbalancer.type=lb
> worker.loadbalancer.balance_workers=node1
> worker.status.type=status
> --
> View this message in context: http://www.nabble.com/Apache-mod_jk-serves-random-files-from-tomcat-tp18385568p18385568.html
> Sent from the Tomcat - User mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: [e-mail]
> To unsubscribe, e-mail: [e-mail]
> For additional commands, e-mail: [e-mail]
>
>

---------------------------------------------------------------------
To start a new topic, e-mail: [e-mail]
To unsubscribe, e-mail: [e-mail]
For additional commands, e-mail: [e-mail]

[Tim Redding]

Hi,

> Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?

All requests are going to tomcat via httpd.

> Does the WAR file get expanded during deployment?

Well tomcat usually extracts the contents to a temporary location so I guess yes.

> Does this happen whether Tomcat is running or not? How about if mod_jk
> is disabled versus enabled?

When tomcat isn't running I get the standard 503 Service Temporarily Unavailable. As this is a production server I can't really afford to disable mod_jk.

We're running mod_jk/1.2.23.  Could probably upgrade. But haven't had an issue with this on our other servers.  I'll wait and see what the accessLogValve presents.

I'm pretty sure the syntax is correct.  It follows the sysntax i've seen in many examples on the net....   I've corrected the case just in case that makes any difference.

Yeah I've just enabled the AccessLogValve so I'll know more about problem soon....

Thanks for your response.

Tim.

Christopher Schultz-2 wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim,

Tim Redding wrote:
| We are experiencing intermittent problems with a particular site that
is not
| returning the correct file that is requested.  For instance if we request
| the index.html file we actually get a css file or even an image.  From the
| apache access log you can see that the size of the index.html file
grows on
| the second request. This is because a gif was actually returned.
|
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] "GET /index.html
HTTP/1.1"
| 200 1068
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] "GET /index.html
HTTP/1.1"
| 200 9526
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] "GET /index.html
HTTP/1.1"
| 200 1086

Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?

Does this happen whether Tomcat is running or not? How about if mod_jk
is disabled versus enabled?

| We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk
| (version unknown but fairly recent).

$ strings /path/to/mod_jk.so | grep mod_jk/
mod_jk/1.2.26

(This is what mine returns). Remember that mod_jk 1.2 has been around
for ... ever and there's a big difference between 1.2.5 and 1.2.26.
Virtually all versions of mod_jk are compatible with all versions of
Apache httpd and Tomcat, so you should upgrade to the latest whenever it
is convenient for you.

| We have all assets in our war file.

Does the WAR file get expanded during deployment?

| JkMount status

Probably not your problem, but is this the correct syntax? I would have
expected something like

JkMount status_uri status

|     jkmount /* loadbalancer

Looks like everything goes to Tomcat. Try enabling the AccessLogValve to
see which requests Tomcat is responding to -- that may help shed a bit
of light on the problem.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkh2eEsACgkQ9CaO5/Lv0PDSZwCgjk7OvOHHAZpvDDolD3JAgIdq
EVgAnRWBvsQbbNZlSvJsRp+b2dLmT0ml
=9ZiZ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: [e-mail]
To unsubscribe, e-mail: [e-mail]
For additional commands, e-mail: [e-mail]

[Rainer Jung-3]

Hi Tim,

Tim Redding schrieb:

> Hi,
>
> We are experiencing intermittent problems with a particular site that is not
> returning the correct file that is requested.  For instance if we request
> the index.html file we actually get a css file or even an image.  From the
> apache access log you can see that the size of the index.html file grows on
> the second request. This is because a gif was actually returned.
>
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] "GET /index.html HTTP/1.1"
> 200 1068
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] "GET /index.html HTTP/1.1"
> 200 9526
> XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] "GET /index.html HTTP/1.1"
> 200 1086

This is pretty serious (I assume that 1086!=1968 was a typo).

> No error messages are logged in the mode_jk.log file.
>
> We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk
> (version unknown but fairly recent).  We have all assets in our war file.
> When we hit Tomcat directly on port 8080 it serves the correct file. And to
> fix the problem an apache restart seems to sort things out.
>
> On this server with have 2 vhosts.  One is a simple nothing fancy static
> site and the other forwards everything to our Tomcat server.  Below I've
> included our mod_jk config and a snippet of our httpd.conf.  
>
> Any ideas or things to try would be most appreciated.

What's you platform and which httpd MPM (prefork orworker or something
else) do you use? For some platforms (e.g. AIX) the detection of
multi-threading in httpd during mpod_jk build-time was broken. Starting
with 1.2.24 we build always including multi-thread support unless
explicitely stated via a configure option. If you 1.2.23 build is not
thread safe, but your httpd uses threads (like with worker mpm), then
such trouble is possible, although more likely you would see crashes
etc. For most platforms like Linux and Solaris the threading detection
was OK already before 1.2.24.

Another possible (but not very likely) cause could be bug 44494 of
Tomcat 6.0.16 which under certain circumstances could leave data in the
request object after request handling completed. You could try either
downgrding to 6.0.15 or upgrading to the soon to be expected 6.0.17.

I would also add the access log on the Tomcat side. If you find the same
phenomenon there, then it's unlikely, that httpd/mod_jk are responsible
and the reason should be inside Tomcat or the webapp.

Can you reproduce the problem on a test system?

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Rainer Jung-3]

Christopher Schultz schrieb:
> | JkMount status
>
> Probably not your problem, but is this the correct syntax?

You can use that syntax in Location directives. Then the missing URL is
taken from the Location URL, so the mount is valid for each URL in the
respective Location.

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[dave.smith]

> Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?

Not too weird.  I am experiencing the same thing with Tomcat 5.5 and mod_jk 1.2.23.  I have Tomcat serving everything.

I am also using a load balancer that sends an OPTION every 2 seconds to each web server to make sure that the server is alive.

This intermittent random response issue is really killing me.

Christopher Schultz-2 wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Tim,

Tim Redding wrote:
| We are experiencing intermittent problems with a particular site that
is not
| returning the correct file that is requested.  For instance if we request
| the index.html file we actually get a css file or even an image.  From the
| apache access log you can see that the size of the index.html file
grows on
| the second request. This is because a gif was actually returned.
|
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:10:39 +0100] "GET /index.html
HTTP/1.1"
| 200 1068
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:10 +0100] "GET /index.html
HTTP/1.1"
| 200 9526
| XXX.XXX.XXX.130 - - [10/Jul/2008:15:13:48 +0100] "GET /index.html
HTTP/1.1"
| 200 1086

Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?

Does this happen whether Tomcat is running or not? How about if mod_jk
is disabled versus enabled?

| We have Apache/2.2.3 on the front on a Tomcat 6.0.16 server with mod_jk
| (version unknown but fairly recent).

$ strings /path/to/mod_jk.so | grep mod_jk/
mod_jk/1.2.26

(This is what mine returns). Remember that mod_jk 1.2 has been around
for ... ever and there's a big difference between 1.2.5 and 1.2.26.
Virtually all versions of mod_jk are compatible with all versions of
Apache httpd and Tomcat, so you should upgrade to the latest whenever it
is convenient for you.

| We have all assets in our war file.

Does the WAR file get expanded during deployment?

| JkMount status

Probably not your problem, but is this the correct syntax? I would have
expected something like

JkMount status_uri status

|     jkmount /* loadbalancer

Looks like everything goes to Tomcat. Try enabling the AccessLogValve to
see which requests Tomcat is responding to -- that may help shed a bit
of light on the problem.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkh2eEsACgkQ9CaO5/Lv0PDSZwCgjk7OvOHHAZpvDDolD3JAgIdq
EVgAnRWBvsQbbNZlSvJsRp+b2dLmT0ml
=9ZiZ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: [e-mail]
To unsubscribe, e-mail: [e-mail]
For additional commands, e-mail: [e-mail]

[Rainer Jung-3]

dave.smith schrieb:
>> Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?
>
> Not too weird.  I am experiencing the same thing with Tomcat 5.5 and mod_jk
> 1.2.23.  I have Tomcat serving everything.
>
> I am also using a load balancer that sends an OPTION every 2 seconds to each
> web server to make sure that the server is alive.
>
> This intermittent random response issue is really killing me.

Could you please also add some info:

Tomcat version?

And from my previous mail:

What's you platform and which httpd MPM (prefork orworker or something
else) do you use? For some platforms (e.g. AIX) the detection of
multi-threading in httpd during mpod_jk build-time was broken. Starting
with 1.2.24 we build always including multi-thread support unless
explicitely stated via a configure option. If you 1.2.23 build is not
thread safe, but your httpd uses threads (like with worker mpm), then
such trouble is possible, although more likely you would see crashes
etc. For most platforms like Linux and Solaris the threading detection
was OK already before 1.2.24.

Another possible (but not very likely) cause could be bug 44494 of
Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data
in the request object after request handling completed. You could try
either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be
expected 6.0.17/5.5.27.

I would also add the access log on the Tomcat side. If you find the same
phenomenon there, then it's unlikely, that httpd/mod_jk are responsible
and the reason should be inside Tomcat or the webapp.

Can you reproduce the problem on a test system?

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[dave.smith]

Hi Rainer,

Thanks a lot for the reply.

I am using Tomcat 5.5.25 (rpm from jpackage.org).  CentOS Linux 2.6.18.

httpd was compiled in prefork mode. The prefork settings are:

StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000

I have setup JMeter to run against a test environment, but was unable to reproduce.  These random responses occur in production about once every week or so/more.  The problem will often (temporarily) correct itself, but sometimes I will need to restart httpd if the problem persists -- restarting tomcat also works to temporarily correct the problem.

The only thing strange that I see in my logs are in the test_client.log:

WARNING: Exception thrown whilst processing POSTed parameters
java.io.IOException: Socket read failed
        at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
        ...

Rainer Jung-3 wrote:

dave.smith schrieb:
>> Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?
>
> Not too weird.  I am experiencing the same thing with Tomcat 5.5 and mod_jk
> 1.2.23.  I have Tomcat serving everything.
>
> I am also using a load balancer that sends an OPTION every 2 seconds to each
> web server to make sure that the server is alive.
>
> This intermittent random response issue is really killing me.

Could you please also add some info:

Tomcat version?

And from my previous mail:

What's you platform and which httpd MPM (prefork orworker or something
else) do you use? For some platforms (e.g. AIX) the detection of
multi-threading in httpd during mpod_jk build-time was broken. Starting
with 1.2.24 we build always including multi-thread support unless
explicitely stated via a configure option. If you 1.2.23 build is not
thread safe, but your httpd uses threads (like with worker mpm), then
such trouble is possible, although more likely you would see crashes
etc. For most platforms like Linux and Solaris the threading detection
was OK already before 1.2.24.

Another possible (but not very likely) cause could be bug 44494 of
Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data
in the request object after request handling completed. You could try
either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be
expected 6.0.17/5.5.27.

I would also add the access log on the Tomcat side. If you find the same
phenomenon there, then it's unlikely, that httpd/mod_jk are responsible
and the reason should be inside Tomcat or the webapp.

Can you reproduce the problem on a test system?

Regards,

Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: [e-mail]
To unsubscribe, e-mail: [e-mail]
For additional commands, e-mail: [e-mail]

[Rainer Jung-3]

Hi David,

dave.smith schrieb:

> Hi Rainer,
>
> Thanks a lot for the reply.
>
> I am using Tomcat 5.5.25 (rpm from jpackage.org).  CentOS Linux 2.6.18.
>
> httpd was compiled in prefork mode. The prefork settings are:
>
> StartServers       8
> MinSpareServers    5
> MaxSpareServers   20
> ServerLimit      256
> MaxClients       256
> MaxRequestsPerChild  4000
>
> I have setup JMeter to run against a test environment, but was unable to
> reproduce.  These random responses occur in production about once every week
> or so/more.  The problem will often (temporarily) correct itself, but
> sometimes I will need to restart httpd if the problem persists -- restarting
> tomcat also works to temporarily correct the problem.
>
> The only thing strange that I see in my logs are in the test_client.log:
>
> WARNING: Exception thrown whilst processing POSTed parameters
> java.io.IOException: Socket read failed
>         at
> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
>         ...

Thanks for the information. What is "test_client.log"? It looks like a
Tomcat log file? Could you also post a larger part of the stack, or do
you only get one line?

Would you be able to do the following two things, maybe not both at the
same time:

- disable the apr connector (tcnative.so)
- upgrade jk to 1.2.26

Concerning the apr connector: If you are using OpenSSL with apr and
Tomcat or you have some similar reason you really need it, then don't
switch. But if you use it without a very specific reason, disabling it
for a week or two would help us isolate the problem.

Concerning mod_jk upgrade: That should be very easy, apart from the
following: if your httpd uses VirtualHost in the configuration, you have
to include your JkMount inside the VirtualHost, not in the global part,
or you add JkMountCopy On to the VirtualHost.

Regards,

Rainer

> Rainer Jung-3 wrote:
>> dave.smith schrieb:
>>>> Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?
>>> Not too weird.  I am experiencing the same thing with Tomcat 5.5 and
>>> mod_jk
>>> 1.2.23.  I have Tomcat serving everything.
>>>
>>> I am also using a load balancer that sends an OPTION every 2 seconds to
>>> each
>>> web server to make sure that the server is alive.
>>>
>>> This intermittent random response issue is really killing me.
>> Could you please also add some info:
>>
>> Tomcat version?
>>
>> And from my previous mail:
>>
>> What's you platform and which httpd MPM (prefork orworker or something
>> else) do you use? For some platforms (e.g. AIX) the detection of
>> multi-threading in httpd during mpod_jk build-time was broken. Starting
>> with 1.2.24 we build always including multi-thread support unless
>> explicitely stated via a configure option. If you 1.2.23 build is not
>> thread safe, but your httpd uses threads (like with worker mpm), then
>> such trouble is possible, although more likely you would see crashes
>> etc. For most platforms like Linux and Solaris the threading detection
>> was OK already before 1.2.24.
>>
>> Another possible (but not very likely) cause could be bug 44494 of
>> Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data
>> in the request object after request handling completed. You could try
>> either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be
>> expected 6.0.17/5.5.27.
>>
>> I would also add the access log on the Tomcat side. If you find the same
>> phenomenon there, then it's unlikely, that httpd/mod_jk are responsible
>> and the reason should be inside Tomcat or the webapp.
>>
>> Can you reproduce the problem on a test system?
>>
>> Regards,
>>
>> Rainer

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[markt-2]

Rainer Jung wrote:
> Hi David,
>
> dave.smith schrieb:
>> Hi Rainer,
>>
>> Thanks a lot for the reply.
>>
>> I am using Tomcat 5.5.25 (rpm from jpackage.org).  CentOS Linux 2.6.18.

Could you be seeing  CVE-2007-6286 ?

See http://tomcat.apache.org/security-5.html for info.

Mark



---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[dave.smith]

Yesterday, I upgraded our dev environment to mod_jk 1.2.26, which couldn't have been easier.  It will probably take me a couple of days before I can get this done in production, though.

I terminate all HTTPS requests before they get to the web server, so from what you have described, it is probably safe to disable the APR connector.  How do I disable it, though?  I will look into disabling this after I have updated mod_jk in production.

Here's the full stack trace for that exception, displayed in my Tomcat logs:

Jul 10, 2008 10:06:50 PM org.apache.catalina.connector.Request parseParameters
WARNING: Exception thrown whilst processing POSTed parameters
java.io.IOException: Socket read failed
        at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
        at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1158)
        at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1090)
        at org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1228)
        at org.apache.coyote.Request.doRead(Request.java:419)
        at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265)
        at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403)
        at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280)
        at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193)
        at org.apache.catalina.connector.Request.readPostBody(Request.java:2400)
        at org.apache.catalina.connector.Request.parseParameters(Request.java:2379)
        at org.apache.catalina.connector.Request.getParameterNames(Request.java:1047)
        at org.apache.catalina.connector.RequestFacade.getParameterNames(RequestFacade.java:369)
        at org.apache.struts.util.RequestUtils.populate(RequestUtils.java:1225)
        at org.apache.struts.action.RequestProcessor.processPopulate(RequestProcessor.java:821)
        at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:254)
        at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1482)
        at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:525)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
        at org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:444)
        at org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:472)
        at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1286)
        at java.lang.Thread.run(Thread.java:595)

Thanks again,
Dave

Rainer Jung-3 wrote:

Hi David,

dave.smith schrieb:
> Hi Rainer,
>
> Thanks a lot for the reply.
>
> I am using Tomcat 5.5.25 (rpm from jpackage.org).  CentOS Linux 2.6.18.
>
> httpd was compiled in prefork mode. The prefork settings are:
>
> StartServers       8
> MinSpareServers    5
> MaxSpareServers   20
> ServerLimit      256
> MaxClients       256
> MaxRequestsPerChild  4000
>
> I have setup JMeter to run against a test environment, but was unable to
> reproduce.  These random responses occur in production about once every week
> or so/more.  The problem will often (temporarily) correct itself, but
> sometimes I will need to restart httpd if the problem persists -- restarting
> tomcat also works to temporarily correct the problem.
>
> The only thing strange that I see in my logs are in the test_client.log:
>
> WARNING: Exception thrown whilst processing POSTed parameters
> java.io.IOException: Socket read failed
>         at
> org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1037)
>         ...

Thanks for the information. What is "test_client.log"? It looks like a
Tomcat log file? Could you also post a larger part of the stack, or do
you only get one line?

Would you be able to do the following two things, maybe not both at the
same time:

- disable the apr connector (tcnative.so)
- upgrade jk to 1.2.26

Concerning the apr connector: If you are using OpenSSL with apr and
Tomcat or you have some similar reason you really need it, then don't
switch. But if you use it without a very specific reason, disabling it
for a week or two would help us isolate the problem.

Concerning mod_jk upgrade: That should be very easy, apart from the
following: if your httpd uses VirtualHost in the configuration, you have
to include your JkMount inside the VirtualHost, not in the global part,
or you add JkMountCopy On to the VirtualHost.

Regards,

Rainer

> Rainer Jung-3 wrote:
>> dave.smith schrieb:
>>>> Wow. That's weird. Is Tomcat serving the file, or is httpd serving it?
>>> Not too weird.  I am experiencing the same thing with Tomcat 5.5 and
>>> mod_jk
>>> 1.2.23.  I have Tomcat serving everything.
>>>
>>> I am also using a load balancer that sends an OPTION every 2 seconds to
>>> each
>>> web server to make sure that the server is alive.
>>>
>>> This intermittent random response issue is really killing me.
>> Could you please also add some info:
>>
>> Tomcat version?
>>
>> And from my previous mail:
>>
>> What's you platform and which httpd MPM (prefork orworker or something
>> else) do you use? For some platforms (e.g. AIX) the detection of
>> multi-threading in httpd during mpod_jk build-time was broken. Starting
>> with 1.2.24 we build always including multi-thread support unless
>> explicitely stated via a configure option. If you 1.2.23 build is not
>> thread safe, but your httpd uses threads (like with worker mpm), then
>> such trouble is possible, although more likely you would see crashes
>> etc. For most platforms like Linux and Solaris the threading detection
>> was OK already before 1.2.24.
>>
>> Another possible (but not very likely) cause could be bug 44494 of
>> Tomcat 6.0.16/5.5.26 which under certain circumstances could leave data
>> in the request object after request handling completed. You could try
>> either downgrading to 6.0.15/5.5.25 or upgrading to the soon to be
>> expected 6.0.17/5.5.27.
>>
>> I would also add the access log on the Tomcat side. If you find the same
>> phenomenon there, then it's unlikely, that httpd/mod_jk are responsible
>> and the reason should be inside Tomcat or the webapp.
>>
>> Can you reproduce the problem on a test system?
>>
>> Regards,
>>
>> Rainer


---------------------------------------------------------------------
To start a new topic, e-mail: [e-mail]
To unsubscribe, e-mail: [e-mail]
For additional commands, e-mail: [e-mail]