« Return to Thread: Cocoon 2.1.10 Application Freezing
Cocoon 2.1.10 Application Freezing
by Christofer Dutz-3
::
Rate this Message:
Some parts of this message have been removed.
Learn more about Nabble's security policy.
Hi,
I am sort of going to get a heart attack in
the next hours, if I can’t resolve this quickly.
We recently deployed a new Version of a
intranet-application of a large logistic company here in Germany. It is
completely written in cocoon and performs extraordinary nicely. Until at one
point the System starts thrashing. CPU Usage climbs from 12% to 80% and no
request are served anymore.
Here a little more details on the Setup:
-
Windows 2003 Server
-
4 GB Ram … 1,5 GB reserved
for the appserver
-
Bea Weblogic 9.2
-
JRockit 1.5 VM
-
MySQL 5.1 DB Backend
During normal operation a thread dump is
about 30k big … during thrashing this increases to 1,5MB.
Yesterday we had similar Problems which
were unfortunately related to too much logging (100 Threads blocked because
they were locked from the Cocoon Log system). After removing a lot of the Log
Output the system performed nicely even under heavier load. Until the whole
thing blew up again. This time the Thread Dump showed a lot of Threads blocked
at:
"[ACTIVE] ExecuteThread: '3' for
queue: 'weblogic.kernel.Default (self-tuning)'" id=35 idx=0x80 tid=5092
prio=5 alive, in native, blocked, daemon
-- Blocked trying to get lock:
java/lang/Object@0x011A8D38[unlocked]
at
jrockit/vm/Threads.waitForUnblockSignal()V(Native Method)
at
jrockit/vm/Locks.fatLockBlockOrSpin(Locks.java:1630)[optimized]
at
jrockit/vm/Locks.lockFat(Locks.java:1731)[optimized]
at
jrockit/vm/Locks.monitorEnterSecondStage(Locks.java:1277)[optimized]
at
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(InstrumentedResourceLimitingPool.java:411)[inlined]
at
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(PoolableComponentHandler.java:212)[optimized]
at
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.java:425)[inlined]
at
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(ExcaliburComponentSelector.java:307)[inlined]
at
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedComponentSelector.java:301)[optimized]
at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.recycle(AbstractProcessingPipeline.java:743)[optimized]
at
org/apache/avalon/excalibur/pool/InstrumentedResourceLimitingPool.put(InstrumentedResourceLimitingPool.java:407)[inlined]
at
org/apache/avalon/excalibur/component/PoolableComponentHandler.doPut(PoolableComponentHandler.java:212)[optimized]
at
org/apache/avalon/excalibur/component/ComponentHandler.put(ComponentHandler.java:425)[inlined]
at
org/apache/avalon/excalibur/component/ExcaliburComponentSelector.release(ExcaliburComponentSelector.java:307)[inlined]
at
org/apache/cocoon/components/ExtendedComponentSelector.release(ExtendedComponentSelector.java:301)[optimized]
at org/apache/cocoon/components/treeprocessor/InvokeContext.dispose(InvokeContext.java:283)
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:179)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTypeNode.java:139)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[optimized]
at
org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]
at
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[optimized]
at
javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[optimized]
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:104)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(CloseHibernateSessionFilter.java:25)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[inlined]
at
weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3242)[optimized]
at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2010)[inlined]
at weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:1916)[optimized]
at
weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1366)[optimized]
at
weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]
at
weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
at jrockit/vm/RNI.c2java(IIIII)V(Native
Method)
-- end of trace
As far as I understand the thing, the
problem are pool limitations to some of the sitemap components. In general this
should not be a problem. Unfortunately my application uses JavaFlow and a lot
of Flows have to send Data to other Systems. They all use internal pipelines to
generate this output. My guess now is, that if a Flow is executed to produce
output, it blocks some resources. Maybe some resources it needs itself to
finish the request, causing a deadlock. This would be the case in the following
Thread-Dump:
"[ACTIVE] ExecuteThread: '4' for
queue: 'weblogic.kernel.Default (self-tuning)'" id=36 idx=0x84 tid=5100
prio=5 alive, in native, native_blocked, daemon
at
jrockit/vm/Allocator.allocLargeObjectOrArray(IIIZ)Ljava/lang/Object;(Native
Method)
at
jrockit/vm/Allocator.allocLargerThanFreeTLA(Allocator.java:831)[inlined]
at
jrockit/vm/Allocator.allocSlowCaseInner(Allocator.java:947)[inlined]
at
jrockit/vm/Allocator.allocSlowCase(Allocator.java:793)[optimized]
at
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java:63)[optimized]
at
org/apache/cocoon/util/BufferedOutputStream.<init>(BufferedOutputStream.java:46)
at
org/apache/cocoon/environment/AbstractEnvironment.getOutputStream(AbstractEnvironment.java:490)
at
org/apache/cocoon/environment/http/HttpEnvironment.getOutputStream(HttpEnvironment.java:263)
at
org/apache/cocoon/environment/wrapper/EnvironmentWrapper.getOutputStream(EnvironmentWrapper.java:311)
at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.processXMLPipeline(AbstractProcessingPipeline.java:577)[optimized]
at
org/apache/cocoon/components/pipeline/AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:481)[optimized]
at
org/apache/cocoon/components/treeprocessor/sitemap/SerializeNode.invoke(SerializeNode.java:121)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.handleCocoonRedirect(ConcreteTreeProcessor.java:299)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.access$000(ConcreteTreeProcessor.java:48)[inlined]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor$TreeProcessorRedirector.cocoonRedirect(ConcreteTreeProcessor.java:340)[inlined]
at
org/apache/cocoon/environment/ForwardRedirector.redirect(ForwardRedirector.java:60)[optimized]
at
org/apache/cocoon/components/flow/java/AbstractContinuable.sendPage(AbstractContinuable.java:99)[optimized]
at
de/transa/disco/flow/im/InstantMessengerFlow.doCheckMessageClosed(InstantMessengerFlow.java:88)
at
sun/reflect/GeneratedMethodAccessor1853.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown
Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at
java/lang/reflect/Method.invoke(Method.java:585)[optimized]
at
org/apache/cocoon/components/flow/java/JavaInterpreter.callFunction(JavaInterpreter.java:173)[optimized]
at
org/apache/cocoon/components/treeprocessor/sitemap/CallFunctionNode.invoke(CallFunctionNode.java:139)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at
org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MountNode.invoke(MountNode.java:118)[optimized]
at org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/ActTypeNode.invoke(ActTypeNode.java:139)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:47)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/MatchNode.invoke(MatchNode.java:108)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelineNode.invoke(PipelineNode.java:143)[optimized]
at
org/apache/cocoon/components/treeprocessor/AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:69)[inlined]
at
org/apache/cocoon/components/treeprocessor/sitemap/PipelinesNode.invoke(PipelinesNode.java:93)[optimized]
at org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:235)[optimized]
at
org/apache/cocoon/components/treeprocessor/ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:177)[inlined]
at org/apache/cocoon/components/treeprocessor/TreeProcessor.process(TreeProcessor.java:253)[optimized]
at
org/apache/cocoon/Cocoon.process(Cocoon.java:699)[optimized]
at
org/apache/cocoon/servlet/CocoonServlet.service(CocoonServlet.java:1154)[optimized]
at javax/servlet/http/HttpServlet.service(HttpServlet.java:856)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)[optimized]
at
weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)[inlined]
at
weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:283)[optimized]
at
weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/filter/SetCharacterEncodingFilter.doFilter(SetCharacterEncodingFilter.java:104)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[optimized]
at
de/transa/disco/database/hibernate/CloseHibernateSessionFilter.doFilter(CloseHibernateSessionFilter.java:25)[optimized]
at
weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:42)[inlined]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3242)[optimized]
at
weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at
weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at
weblogic/servlet/internal/WebAppServletContext.securedExecute(WebAppServletContext.java:2010)[inlined]
at
weblogic/servlet/internal/WebAppServletContext.execute(WebAppServletContext.java:1916)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.run(ServletRequestImpl.java:1366)[optimized]
at
weblogic/work/ExecuteThread.execute(ExecuteThread.java:209)[optimized]
at
weblogic/work/ExecuteThread.run(ExecuteThread.java:181)
at jrockit/vm/RNI.c2java(IIIII)V(Native
Method)
-- end of trace
Causing the System to immediately run out
of resources.
The only solution I can think of would be
to turn off pooling of components. Where and how can I do this? Is this simply
done by setting the pool-max properties in the core.properties to 0 or do I
have to do more?
We currently have 4 CPUs and 4GB Ram, but
this could be extended if we need to (Virtual environment).
Regards,
Chris
[ C h
r i s t o f e r D u t z ]
C-Ware IT-Service
Inhaber
Dipl. Inf. Christofer Dutz
Karlstraße. 104, 64285 Darmstadt
fon: 0 61 51 / 27315 - 61
fax: 0 61 51 / 27315 - 64
mobil: 0171 / 7 444 2 33
email: christofer.dutz@...
http://www.c-ware.de
FA Darmstadt: 07 813 60581
« Return to Thread: Cocoon 2.1.10 Application Freezing
| Free Forum Powered by Nabble | Forum Help |
