Help Requested: Sending file attachments greater than 4mb causes OOM errors

[Lyman-7]

I appreciate any help or guidance you may provide. I am trying to determine if I am trying to solve a solvable problem or should I pursue other methods to overcome the problem.  

Particulars:
Apache/1.3.27 Ben-SSL/1.48 (Unix) Debian GNU/Linux mod_jk/1.2.5 mod_perl/1.27
tomcat4_4.1.28-1
Axis 1.1
java version "1.4.2_02"
Linux version 2.4.22  (gcc version 2.95.4 20011002 (Debian prerelease)) #1 Wed Oct 1 14:41:07 EDT 2003


Overview: Sending data to a external service, sometimes the data includes file attachments. The external service requested we convert from a get file attachments (sending a url for the file) to a push file attachments process.  The conversion has been working just fine but when we tried to send a file larger than 4 mb we started getting java.lang.OutOfMemoryError. I increased the machine specs from 64mb ram 500 mhz pII to a P4 1.3ghz 1gb ram. Error now takes much less time to manifest itself but still occurs. The error usually occurs when tomcat is in this statement:

java.lang.Object _resp = _call.invoke(new java.lang.Object[] {pvarSspNumber, pvarContractNumber, pnumIdrNumber, pvarRevision, pvarUrgency, pvarItemNumber, pvarDescription, pvarIsPartial, pvarWorkItemParagraph, pvarReportType, pvarRecommendation, pvarAnswerMandatory, pvarReportAction, pvarResponseReqByDt, pKtrCertificateNm, pvarBriefTx, pvarKtrMatl, pvarKtrCtLastNm, pvarLocation, pnumCostProposal, pvarKtrComments, ptypAttachMetaBlob});

Using tcpdump, I have seen that the error occurs before sending any packets to the outside service.

I have reviewed the archives and found much talk about sending files with soap messages and how the file size is usually restricted.  I have also seen talk of being able to send 20mb files. The outside service confirms they are able to send 9mb files using .net routines.

My question is this I guess, is it reasonable to think I should be able to send files larger than 4mb.  If it is reasonable, then what can I do to accomplish that.  Those that are sending 20mb files, what did you have to do to be able to accomplish the task?

Again, I am most appreciative of any assistance in resolving the problem. If anyone on the list would be able to help me have the capability to send larger files but would like to be paid for that assistance, please contact me off list.

This is my first post to the list so thanks for you patience and understanding if I fall short of providing the information needed to provide help.

Lyman

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Peter Crowther]

> From: Lyman [mailto:lbyrd@...]
> I increased the machine specs
> from 64mb ram 500 mhz pII to a P4 1.3ghz 1gb ram.

Java will only use the amount of RAM you tell it to, not the entire machine RAM.  It defaults to a low value - probably 64Mbytes in your environment.  You'll need to add appropriate statements to set Java's minimum and maximum heap memory to your Tomcat startup script.  From memory (it's a long time since I used a non-Windows Tomcat) this comes down to adding something like:

export JAVA_OPTS=-Xms512m -Xmx768m

to catalina.sh - if it already has JAVA_OPTS in there, just add those to the end.  You'll need to tune them so you still have enough memory for the OS and any other services on the box.

http://wiki.apache.org/tomcat/FAQ/Memory is probably useful reading.

                - Peter

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Lyman-7]

Thank you Peter, I forgot to add that I had set the parameters as follows:
"-Djava.awt.headless=true -Xmx1500m -Xms1500m"

The -D parameter was there before so I did not mess with it.. I had tried with and without the -Xms parameter and did not see any difference in behavior.

Lyman


Peter Crowther wrote:

>> From: Lyman [mailto:lbyrd@...]
>> I increased the machine specs
>> from 64mb ram 500 mhz pII to a P4 1.3ghz 1gb ram.
>
> Java will only use the amount of RAM you tell it to, not the entire machine RAM.  It defaults to a low value - probably 64Mbytes in your environment.  You'll need to add appropriate statements to set Java's minimum and maximum heap memory to your Tomcat startup script.  From memory (it's a long time since I used a non-Windows Tomcat) this comes down to adding something like:
>
> export JAVA_OPTS=-Xms512m -Xmx768m
>
> to catalina.sh - if it already has JAVA_OPTS in there, just add those to the end.  You'll need to tune them so you still have enough memory for the OS and any other services on the box.
>
> http://wiki.apache.org/tomcat/FAQ/Memory is probably useful reading.
>
>                 - Peter
>
> ---------------------------------------------------------------------
> To start a new topic, e-mail: users@...
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>
>

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Peter Crowther]

> From: Lyman [mailto:lbyrd@...]
> Thank you Peter, I forgot to add that I had set the
> parameters as follows:
> "-Djava.awt.headless=true -Xmx1500m -Xms1500m"

On a machine with 1 Gbyte of physical memory, that is almost certainly a mistake - depending on how much swap you've allocated.  Have you tried with smaller values, such as 50% to 75% of physical memory?

> The -D parameter was there before so I did not mess with it..

Means it's possible to render images with AWT without requiring a local display.

> I had tried with and without the -Xms parameter and did not
> see any difference in behavior.

What does "top" tell you about your used/free memory - total, and that used by Java?

                - Peter

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Lyman-7]

Peter Crowther wrote:
>
> What does "top" tell you about your used/free memory - total, and that used by Java?
>

Here is a snap shot of top sorted by %memory used:
top - 11:02:53 up 12 days,  2:56,  1 user,  load average: 0.02, 0.01, 0.00
Tasks:  76 total,   1 running,  75 sleeping,   0 stopped,   0 zombie
Cpu(s):   0.0% user,   1.6% system,   0.0% nice,  98.4% idle
Mem:    904948k total,   892732k used,    12216k free,    86580k buffers
Swap:   248996k total,     6988k used,   242008k free,   643228k cached

and for the tomcat processes:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
29154 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:14.59 java                  
29155 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.46 java                  
29156 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:04.03 java                  
29157 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.00 java                  
29158 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.01 java                  
29159 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.00 java                  
29160 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.00 java                  
29161 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:01.25 java                  
29162 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:19.77 java                  
29163 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.08 java                  
29164 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.00 java                  
29165 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.01 java                  
29166 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.05 java                  
29167 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.04 java                  
29168 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.01 java                  
29169 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.00 java                  
29170 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.00 java                  
29172 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.00 java                  
29173 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.60 java                  
29174 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.02 java                  
29175 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.03 java                  
29176 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.04 java                  
29177 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.03 java                  
29178 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:01.31 java                  
29179 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:08.42 java                  
30161 tomcat4    9   0 92076  89m  13m S  0.0 10.2   0:00.01 java    


I have tried changing -Xmx in increments of 256m but not seen any difference in behavior there either...

Thanks

Lyman
     

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lyman,

Lyman wrote:
| Thank you Peter, I forgot to add that I had set the parameters as follows:
| "-Djava.awt.headless=true -Xmx1500m -Xms1500m"

On a 1gb machine? I'm not sure Java will even start up with those
parameters. Are you sure your memory values are being respected?

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgrG68ACgkQ9CaO5/Lv0PDYcgCfb/Cn+19/5CgNQOsZpYcZYwPc
TdEAn1wYWK2UFUqeOPloZf88wmpOoKQY
=Flpf
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Christopher Schultz-2]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lyman,

Lyman wrote:
| mod_jk/1.2.5

You should upgrade this if possible. That version is 5 years old and the
latest version should be completely compatible with the old one. I'm
sure it's not the problem, but I wouldn't want to miss an opportunity to
suggest an upgrade. Lots of good stuff in later releases.

| The error usually occurs when tomcat is in this statement:
|
| java.lang.Object _resp = _call.invoke(new java.lang.Object[]
| {pvarSspNumber, pvarContractNumber, pnumIdrNumber, pvarRevision,
| pvarUrgency, pvarItemNumber, pvarDescription, pvarIsPartial,
| pvarWorkItemParagraph, pvarReportType, pvarRecommendation,
| pvarAnswerMandatory, pvarReportAction, pvarResponseReqByDt,
| pKtrCertificateNm, pvarBriefTx, pvarKtrMatl, pvarKtrCtLastNm,
| pvarLocation, pnumCostProposal, pvarKtrComments, ptypAttachMetaBlob});

Any indication of what "_call" is referring to? That's obviously an
instance of java.lang.reflect.Method, so what method is it pointing to?
What does that method do?

| Using tcpdump, I have seen that the error occurs before sending any
| packets to the outside service.

What type of message-passing are you using? SOAP? XML-RPC? REST? Are you
using a particular library? Looks like Axis. Have you asked the Axis
folks about this?

For testing purposes, could you use a more recent JVM? Sun's Java 1.6
has an option for emitting a heap dump whenever an OOME occurs:

- -XX:+HeapDumpOnOutOfMemoryError

It's possible that your application simply needs more memory than it has
been configured to use. I saw your 1.5GB heap configuration, but it
would be good to verify that the JVM is actually respecting that by
observing the memory values within the running JVM.

You can write yourself a little JSP that dumps memory information from
the java.lang.Runtime object (or just use the one I wrote called
SessionSnooper which you can find here:
http://www.christopherschultz.net/projects/java/), or you can install
something like Lambda Probe (it's a webapp -- easy to install) which
gives you a ton of other information, too. Just make sure that your heap
isn't small like 64MB or something like that. Your 'top' output would
suggest that the heap is larger, though.

- -chris
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkgrH/8ACgkQ9CaO5/Lv0PBWlwCgp4no45qxw5IiYA+vDWitS3C6
fmEAn0g7LMo04NWfvR5fj3xM9LW8jgK1
=6MD1
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Lyman-7]

Sorry to take so long getting back to you and thanks for the post.. others pointed out this is probably an axis problem so I have been searching the archives of the axis mailing list for clues.

Christopher Schultz wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
>
> You should upgrade this if possible. That version is 5 years old and the
> latest version should be completely compatible with the old one. I'm
> sure it's not the problem, but I wouldn't want to miss an opportunity to
> suggest an upgrade. Lots of good stuff in later releases.

Yes I agree. I just wanted to get this problem resolved before I threw more into the mix unless someone is sure that upgrading will resolve the problem.


>
> Any indication of what "_call" is referring to? That's obviously an
> instance of java.lang.reflect.Method, so what method is it pointing to?
> What does that method do?

That is an axis method that actually sends the soap message (I think).
 
> What type of message-passing are you using? SOAP? XML-RPC? REST? Are you
> using a particular library? Looks like Axis. Have you asked the Axis
> folks about this?

yes soap and axis (1.1)


>
> For testing purposes, could you use a more recent JVM? Sun's Java 1.6
> has an option for emitting a heap dump whenever an OOME occurs:
>
> - -XX:+HeapDumpOnOutOfMemoryError
>

Thanks for the suggestion.

> It's possible that your application simply needs more memory than it has
> been configured to use. I saw your 1.5GB heap configuration, but it
> would be good to verify that the JVM is actually respecting that by
> observing the memory values within the running JVM.
>
> You can write yourself a little JSP that dumps memory information from
> the java.lang.Runtime object (or just use the one I wrote called
> SessionSnooper which you can find here:
> http://www.christopherschultz.net/projects/java/), or you can install
> something like Lambda Probe (it's a webapp -- easy to install) which
> gives you a ton of other information, too. Just make sure that your heap
> isn't small like 64MB or something like that. Your 'top' output would
> suggest that the heap is larger, though.
>
> - -chris

These are also a very good suggestions, I will try them out. Interesting thing on the top output, I had originally tried increasing the -Xmx value in increments of 256 and found no change in the top output or results of the run so I am not sure it is using the parameters but by all I can tell from the start scripts, it should be. maybe the Runtime object will provide some clues there. I may just try hard coding the parameter in the script that actually executes the java command and see if that makes a difference...

Thanks again for the post.

Lyman

---------------------------------------------------------------------
To start a new topic, e-mail: users@...
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...