How to loop over multiple tiles and get raster data without memory overrun?

[jai-interest-2]

I have a heap allocation problem processing a large, composite image, which is a PlannarImage constructed from a mosaic of four (2x2 588Mb) files. I tile the image into Nx and Ny tiles and loop through all tiles, each time getting the data:

   tileBuffer (which is a DataBuffer) = mosaicImage.getTile(i, j).getDataBuffer();    // i and j are tile indexes
   tileData   (which is byte[] array)   = tileBuffer.getData();

The tile size is only about 1000x1000 pixels, but as I loop through the tile, the allocated memory constantly increases, till eventually I get the heap allocation problem.
The problem is clearly caused by JVM constantly creating a new raster for each tile and using the garbage disposal when it pleases. not when it is needed. I can probably circumvent this problem by increasing heap allocation, or decreasing the input file size. For example processing a mosaic file composed of four 256Mb files is not  a problem on tile size that is over 8k x 8k pixels wide (I have 4GB RAM on my machine)

So my question is how to handle really big mosaic files composed from for example 20x20 2GB files? I would like to be able to fetch data from the raster corresponding to tiles that are 2000x2000 pixels. Clearly, one should be able to handle such cases as long as I have enough storage to accommodate data for one tile (plus some overhead).

Thank you
Gregory
[Message sent by forum member 'gmiecznik' (gmiecznik)]

http://forums.java.net/jive/thread.jspa?messageID=288840

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@...
For additional commands, e-mail: interest-help@...

[Bob Deen]

The JVM will GC before issuing an out of memory error, so that shouldn't
be the problem.

It might have to do with the tile cache.  There is a default JAI tile
cache which holds all tiles computed, up to some limit.  It may not
flush those in low-memory conditions.  So, every tile you request from
mosaicImage should be going in the cache.

Try setting the cache size down (possibly even 0), or attach a custom
cache (size 0?) to the mosaic operator to disable only that one cache.

-Bob

jai-interest@... wrote:

> I have a heap allocation problem processing a large, composite image, which is a PlannarImage constructed from a mosaic of four (2x2 588Mb) files. I tile the image into Nx and Ny tiles and loop through all tiles, each time getting the data:
>
>    tileBuffer (which is a DataBuffer) = mosaicImage.getTile(i, j).getDataBuffer();    // i and j are tile indexes
>    tileData   (which is byte[] array)   = tileBuffer.getData();
>
> The tile size is only about 1000x1000 pixels, but as I loop through the tile, the allocated memory constantly increases, till eventually I get the heap allocation problem.
> The problem is clearly caused by JVM constantly creating a new raster for each tile and using the garbage disposal when it pleases. not when it is needed. I can probably circumvent this problem by increasing heap allocation, or decreasing the input file size. For example processing a mosaic file composed of four 256Mb files is not  a problem on tile size that is over 8k x 8k pixels wide (I have 4GB RAM on my machine)
>
> So my question is how to handle really big mosaic files composed from for example 20x20 2GB files? I would like to be able to fetch data from the raster corresponding to tiles that are 2000x2000 pixels. Clearly, one should be able to handle such cases as long as I have enough storage to accommodate data for one tile (plus some overhead).
>
> Thank you
> Gregory
> [Message sent by forum member 'gmiecznik' (gmiecznik)]
>
> http://forums.java.net/jive/thread.jspa?messageID=288840
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: interest-unsubscribe@...
> For additional commands, e-mail: interest-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@...
For additional commands, e-mail: interest-help@...

[jai-interest-2]

Thanks Bob

I think the problem is caused by  GeoTiff reader by GeoTools. I will dig into that.

Gregory
[Message sent by forum member 'gmiecznik' (gmiecznik)]

http://forums.java.net/jive/thread.jspa?messageID=289148

---------------------------------------------------------------------
To unsubscribe, e-mail: interest-unsubscribe@...
For additional commands, e-mail: interest-help@...