|

»

»

»

JDOM output buffering question

View: New views

11 Messages — Rating Filter: Alert me

	JDOM output buffering question by Gamble, Wesley (WG10) :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Some parts of this message have been removed. Learn more about Nabble's security policy. All, I am looking at generating some big-a**ed XML documents, which I then write out to a file. I seem to be having some issue with the document memory not being released. In order to output the entire document using an XMLOutputter, the entire document must be resident in memory, correct? What I’d like is a way to output using the XMLOutputter periodically so that I could use a fraction of the memory required to hold the entire document. However, I don’t believe that this is truly possible. It seems like if I wanted to do this, I would have to write out all of the contents of the root element using an Outputter and then “wrap” that XML in my root element “by hand.” (meaning using string and file manipulation to handle it. I’m not talking about the buffering of the call to output an entire document, I’m talking about using a “window” in memory to hold parts of the document while I continuously write it to the filesystem, so that I don’t need to build the entire document in memory. Hope that makes sense. Thanks for any advice, Wes _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	Re: JDOM output buffering question by jhunter :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Your understanding is correct. I've had occasion to output something akin to a long log file, and it worked well to manually write the root element start tag to the stream, then call XMLOutputter repeatedly to output the various elements, then manually close with the root element end tag. -jh- Gamble, Wesley (WG10) wrote: > All, > > > > I am looking at generating some big-a**ed XML documents, which I then > write out to a file. I seem to be having some issue with the document > memory not being released. > > > > In order to output the entire document using an XMLOutputter, the entire > document must be resident in memory, correct? > > > > What I’d like is a way to output using the XMLOutputter periodically so > that I could use a fraction of the memory required to hold the entire > document. However, I don’t believe that this is truly possible. It > seems like if I wanted to do this, I would have to write out all of the > contents of the root element using an Outputter and then “wrap” that XML > in my root element “by hand.” (meaning using string and file > manipulation to handle it. > > > > I’m not talking about the buffering of the call to output an entire > document, I’m talking about using a “window” in memory to hold parts of > the document while I continuously write it to the filesystem, so that I > don’t need to build the entire document in memory. > > > > Hope that makes sense. > > > > Thanks for any advice, > > Wes > > > ------------------------------------------------------------------------ > > _______________________________________________ > To control your jdom-interest membership: > http://www.jdom.org/mailman/options/jdom-interest/youraddr@... _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	RE: JDOM output buffering question by Gamble, Wesley (WG10) :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Some parts of this message have been removed. Learn more about Nabble's security policy. The reason that I asked my original question was because I was seeing my memory increase by 200MB and then not get released when my document was written, and the method that created it exited. I came upon this thread from a few years ago: http://osdir.com/ml/java.jdom.general/2005-11/msg00041.html which claims that the SAXBuilder is holding onto a reference that keeps the document memory from being freed. Can anyone corroborate this and explain a workaround or JDOM upgrade that might work? We are on JDOM 1.0 - is this resolved in 1.1? The statement in that thread that says "The document won't be available for garbage collection until the next parse begins." is interesting to me - I am never parsing, just writing. So maybe this is also a bug on the "way out" as well as on the "way in"? Also, I didn't quite understand this statement (from Elliotte's reply): "Even if you can't set it to null, but you could set it to a different ContentHandler that doesn't have a reference to anything. DefaultHandler would work nicely for this." It seems like my trouble may stem from keeping the SAXBuilder object around as a class variable. What if I allow my SAXBuilder to be an instance variable, won't its garbage collection allow for the garbage collection of everything it's holding? Thanks, Wes _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	RE: JDOM output buffering question by Gamble, Wesley (WG10) :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Some parts of this message have been removed. Learn more about Nabble's security policy. I put a finalize method on the object that holds the SAXBuilder as an instance variable, and set the SAXBuilder to null, and no memory is released. The descendant of the JDOM Document gets garbage collected, and I’m assuming the SAXBuilder will get garbage collected, but my memory is still way high. Can anyone think of any ways that resources could still be held on to in this scenario. Perhaps my leak is not in JDOM. Thanks, Wes _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	Re: JDOM output buffering question by Rolf Lear-2 :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Gamble, Wesley (WG10) wrote: > > I put a finalize method on the object that holds the SAXBuilder as an > instance variable, and set the SAXBuilder to null, and no memory is > released. > > The descendant of the JDOM Document gets garbage collected, and I’m > assuming the SAXBuilder will get garbage collected, but my memory is > still way high. > > Can anyone think of any ways that resources could still be held on to > in this scenario. Perhaps my leak is not in JDOM. > > Thanks, > > Wes > > ------------------------------------------------------------------------ > > _______________________________________________ > To control your jdom-interest membership: > http://www.jdom.org/mailman/options/jdom-interest/youraddr@... How are you measuring your memory usage? Rolf _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	RE: JDOM output buffering question by Gamble, Wesley (WG10) :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message I'm watching the memory for the Java process in the Windows task manager. -----Original Message----- From: Rolf Lear [mailto:jdom@...] Sent: Tuesday, January 15, 2008 6:41 PM To: Gamble, Wesley (WG10) Cc: jdom-interest@... Subject: Re: [jdom-interest] JDOM output buffering question Gamble, Wesley (WG10) wrote: > > I put a finalize method on the object that holds the SAXBuilder as an > instance variable, and set the SAXBuilder to null, and no memory is > released. > > The descendant of the JDOM Document gets garbage collected, and I'm > assuming the SAXBuilder will get garbage collected, but my memory is > still way high. > > Can anyone think of any ways that resources could still be held on to > in this scenario. Perhaps my leak is not in JDOM. > > Thanks, > > Wes > > ------------------------------------------------------------------------ > > _______________________________________________ > To control your jdom-interest membership: > http://www.jdom.org/mailman/options/jdom-interest/youraddr@... How are you measuring your memory usage? Rolf _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	Re: JDOM output buffering question by Rolf Lear-2 :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message JVM's are typically reluctant to release memory back to the OS. A better way to monitor the actual memory usage (in your case) is to turn on the -verbose:gc commandline-option (or better, the -verbose:gc -XX:PrintGCTimeStamps) to your Java process. This will track your memory usage better, and tell you how much memory gets cleaned up from no-longer-referenced Java classes (Garbage Collected). May I suggest the following reading material: http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepaper.pdf Once Java requests memory from the OS, it is very unlikely to give it back. Whether it gets used (again) or not. Tracking memory from the OS perspective gives you no indication as to whether objects are GC's or not. Another Alternative is to generate a Full heap dump from your JVM (Press Ctrl-ScrollLock in Windows, and Ctrl-\ on Unix (or kill-3 <pid>). Rolf Gamble, Wesley (WG10) wrote: > I'm watching the memory for the Java process in the Windows task > manager. > > -----Original Message----- > From: Rolf Lear [mailto:jdom@...] > Sent: Tuesday, January 15, 2008 6:41 PM > To: Gamble, Wesley (WG10) > Cc: jdom-interest@... > Subject: Re: [jdom-interest] JDOM output buffering question > > Gamble, Wesley (WG10) wrote: > >> I put a finalize method on the object that holds the SAXBuilder as an >> instance variable, and set the SAXBuilder to null, and no memory is >> released. >> >> The descendant of the JDOM Document gets garbage collected, and I'm >> assuming the SAXBuilder will get garbage collected, but my memory is >> still way high. >> >> Can anyone think of any ways that resources could still be held on to >> in this scenario. Perhaps my leak is not in JDOM. >> >> Thanks, >> >> Wes >> >> >> > ------------------------------------------------------------------------ > >> _______________________________________________ >> To control your jdom-interest membership: >> >> > http://www.jdom.org/mailman/options/jdom-interest/youraddr@... > How are you measuring your memory usage? > > Rolf > > _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	Re: JDOM output buffering question by Joe Bowbeer :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message On Jan 21, 2008 8:21 AM, Gamble, Wesley (WG10) <WG10@...> wrote: > I'm watching the memory for the Java process in the Windows task > manager. > NetBean's free profiler is excellent for this: http://www.netbeans.org/features/java/profiler.html _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	RE: JDOM output buffering question by Gamble, Wesley (WG10) :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Rolf, Thanks for this. I may not have a problem at all. I've been away from the Java world for the while. Not sure if I was always unaware of this or just forgot it. Will check with the verbose gc display to determine what's happening. Thanks, Wes -----Original Message----- From: Rolf Lear [mailto:jdom@...] Sent: Monday, January 21, 2008 1:14 PM To: Gamble, Wesley (WG10) Cc: jdom-interest@... Subject: Re: [jdom-interest] JDOM output buffering question JVM's are typically reluctant to release memory back to the OS. A better way to monitor the actual memory usage (in your case) is to turn on the -verbose:gc commandline-option (or better, the -verbose:gc -XX:PrintGCTimeStamps) to your Java process. This will track your memory usage better, and tell you how much memory gets cleaned up from no-longer-referenced Java classes (Garbage Collected). May I suggest the following reading material: http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepap er.pdf Once Java requests memory from the OS, it is very unlikely to give it back. Whether it gets used (again) or not. Tracking memory from the OS perspective gives you no indication as to whether objects are GC's or not. Another Alternative is to generate a Full heap dump from your JVM (Press Ctrl-ScrollLock in Windows, and Ctrl-\ on Unix (or kill-3 <pid>). Rolf Gamble, Wesley (WG10) wrote: > I'm watching the memory for the Java process in the Windows task > manager. > > -----Original Message----- > From: Rolf Lear [mailto:jdom@...] > Sent: Tuesday, January 15, 2008 6:41 PM > To: Gamble, Wesley (WG10) > Cc: jdom-interest@... > Subject: Re: [jdom-interest] JDOM output buffering question > > Gamble, Wesley (WG10) wrote: > >> I put a finalize method on the object that holds the SAXBuilder as an >> instance variable, and set the SAXBuilder to null, and no memory is >> released. >> >> The descendant of the JDOM Document gets garbage collected, and I'm >> assuming the SAXBuilder will get garbage collected, but my memory is >> still way high. >> >> Can anyone think of any ways that resources could still be held on to >> in this scenario. Perhaps my leak is not in JDOM. >> >> Thanks, >> >> Wes >> >> >> > ------------------------------------------------------------------------ > >> _______________________________________________ >> To control your jdom-interest membership: >> >> > http://www.jdom.org/mailman/options/jdom-interest/youraddr@... > How are you measuring your memory usage? > > Rolf > > _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	RE: JDOM output buffering question by Gamble, Wesley (WG10) :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message FYI, I'm on JRE 1.3.1_09. After reviewing the verbose:gc output on my running process, I see that tiny amounts of memory are being reclaimed multiple times/sec., but when I complete generating my JDOM document and writing it out, I do not see the big drop in heap size that I would expect. There must be something I'm holding onto that I don't realize. One question - is GC really happening once for each "[GC.." output line in the verbose output? It looks like it's happening many times/second. Wes -----Original Message----- From: Rolf Lear [mailto:jdom@...] Sent: Monday, January 21, 2008 1:14 PM To: Gamble, Wesley (WG10) Cc: jdom-interest@... Subject: Re: [jdom-interest] JDOM output buffering question JVM's are typically reluctant to release memory back to the OS. A better way to monitor the actual memory usage (in your case) is to turn on the -verbose:gc commandline-option (or better, the -verbose:gc -XX:PrintGCTimeStamps) to your Java process. This will track your memory usage better, and tell you how much memory gets cleaned up from no-longer-referenced Java classes (Garbage Collected). May I suggest the following reading material: http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepap er.pdf Once Java requests memory from the OS, it is very unlikely to give it back. Whether it gets used (again) or not. Tracking memory from the OS perspective gives you no indication as to whether objects are GC's or not. Another Alternative is to generate a Full heap dump from your JVM (Press Ctrl-ScrollLock in Windows, and Ctrl-\ on Unix (or kill-3 <pid>). Rolf Gamble, Wesley (WG10) wrote: > I'm watching the memory for the Java process in the Windows task > manager. > > -----Original Message----- > From: Rolf Lear [mailto:jdom@...] > Sent: Tuesday, January 15, 2008 6:41 PM > To: Gamble, Wesley (WG10) > Cc: jdom-interest@... > Subject: Re: [jdom-interest] JDOM output buffering question > > Gamble, Wesley (WG10) wrote: > >> I put a finalize method on the object that holds the SAXBuilder as an >> instance variable, and set the SAXBuilder to null, and no memory is >> released. >> >> The descendant of the JDOM Document gets garbage collected, and I'm >> assuming the SAXBuilder will get garbage collected, but my memory is >> still way high. >> >> Can anyone think of any ways that resources could still be held on to >> in this scenario. Perhaps my leak is not in JDOM. >> >> Thanks, >> >> Wes >> >> >> > ------------------------------------------------------------------------ > >> _______________________________________________ >> To control your jdom-interest membership: >> >> > http://www.jdom.org/mailman/options/jdom-interest/youraddr@... > How are you measuring your memory usage? > > Rolf > > _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...
	Re: JDOM output buffering question by Rolf Lear-2 :: Rate this Message: Reply to Author \| View Threaded \| Show Only this Message Yes, GC really happens each time .... As for your un-released memory, make sure you close and de-reference all Streams and Readers/Writers, as well as all set all referenced JDom objects to null. Rolf Gamble, Wesley (WG10) wrote: > FYI, I'm on JRE 1.3.1_09. > > After reviewing the verbose:gc output on my running process, I see that > tiny amounts of memory are being reclaimed multiple times/sec., but when > I complete generating my JDOM document and writing it out, I do not see > the big drop in heap size that I would expect. There must be something > I'm holding onto that I don't realize. > > One question - is GC really happening once for each "[GC.." output line > in the verbose output? It looks like it's happening many times/second. > > Wes > > -----Original Message----- > From: Rolf Lear [mailto:jdom@...] > Sent: Monday, January 21, 2008 1:14 PM > To: Gamble, Wesley (WG10) > Cc: jdom-interest@... > Subject: Re: [jdom-interest] JDOM output buffering question > > JVM's are typically reluctant to release memory back to the OS. A better > > way to monitor the actual memory usage (in your case) is to turn on the > -verbose:gc commandline-option (or better, the -verbose:gc > -XX:PrintGCTimeStamps) to your Java process. This will track your memory > > usage better, and tell you how much memory gets cleaned up from > no-longer-referenced Java classes (Garbage Collected). > > May I suggest the following reading material: > http://java.sun.com/j2se/reference/whitepapers/memorymanagement_whitepap > er.pdf > > Once Java requests memory from the OS, it is very unlikely to give it > back. Whether it gets used (again) or not. Tracking memory from the OS > perspective gives you no indication as to whether objects are GC's or > not. > > Another Alternative is to generate a Full heap dump from your JVM (Press > > Ctrl-ScrollLock in Windows, and Ctrl-\ on Unix (or kill-3 <pid>). > > Rolf > > Gamble, Wesley (WG10) wrote: > >> I'm watching the memory for the Java process in the Windows task >> manager. >> >> -----Original Message----- >> From: Rolf Lear [mailto:jdom@...] >> Sent: Tuesday, January 15, 2008 6:41 PM >> To: Gamble, Wesley (WG10) >> Cc: jdom-interest@... >> Subject: Re: [jdom-interest] JDOM output buffering question >> >> Gamble, Wesley (WG10) wrote: >> >> >>> I put a finalize method on the object that holds the SAXBuilder as an >>> > > >>> instance variable, and set the SAXBuilder to null, and no memory is >>> released. >>> >>> The descendant of the JDOM Document gets garbage collected, and I'm >>> assuming the SAXBuilder will get garbage collected, but my memory is >>> still way high. >>> >>> Can anyone think of any ways that resources could still be held on to >>> > > >>> in this scenario. Perhaps my leak is not in JDOM. >>> >>> Thanks, >>> >>> Wes >>> >>> >>> >>> > ------------------------------------------------------------------------ > >> >> >>> _______________________________________________ >>> To control your jdom-interest membership: >>> >>> >>> > http://www.jdom.org/mailman/options/jdom-interest/youraddr@... > >> How are you measuring your memory usage? >> >> Rolf >> >> >> > > > _______________________________________________ To control your jdom-interest membership: http://www.jdom.org/mailman/options/jdom-interest/youraddr@...

LightInTheBox - Buy quality products at wholesale price