Bug 819 – Clean Garbage command is not freeing memory during macro

Bug 819 - Clean Garbage command is not freeing memory during macro


Summary:	Clean Garbage command is not freeing memory during macro

Status:	RESOLVED FIXED

Product:	Fiji
Classification:	Unclassified
Component:	Plugins
Version:	unspecified
Hardware:	PC Windows

Importance:	P4 normal
Assigned To:	ImageJ Bugs Mailing List

URL:

Duplicates:	863 884
Depends on:
Blocks:

Reported:	2014-06-25 13:21 CDT by amc910
Modified:	2014-08-08 14:55 CDT
CC List:	7 users (show)

See Also:

Description amc910 2014-06-25 13:21:00 CDT

Hi there,

I'm running a macro (included below) to create a montage of images from separate folders.
The macro iterates over 24 folders, opening the nth image from each in Fiji (the 21st and 23rd image run were saved with a different naming format, hence the 'if'). It then turns these images into a stack, before making a 5x5 montage of the images, saving this montage, and then closing both the stack and the montage. It then does the same for the n+1th image, and so on.

The issue is that this macro seems to be causing Fiji's memoy to run out after ~ 30-40 loops (montages).
The Collect Garbage command doesn't seem to release this memory. It's also not being released when Fiji is closed, an imageJ service remains running in the Task Manager with ~13,000,000k Memory, which I need to stop manually.

If you need any more information, let me know.

Also, if there's a simpler way to run my macro than I'm using, I'd love to hear it.

Kind regards,
Angus


for(image=1; image<=349; image++){
	for(drop=1; drop<=24; drop++){
		if(drop==21 || drop==23){
			open("F:\\Timelapse\\GlassReg\\GlassShards3\\"+drop+"\\"+IJ.pad(image,3)+".tif");
			}
		else{
			open("F:\\Timelapse\\GlassReg\\GlassShards3\\"+drop+"\\"+IJ.pad(image,5)+".tif");
			}
	}
	run("Images to Stack", "name=Filler title=[] use");
	run("Make Montage...", "columns=5 rows=5 scale=1 first=1 last=24 	increment=1 border=0 font=12");
	run("Save", "save=F:\\Timelapse\\GlassReg\\Full\\" + image + ".tif");
	run("Close");
	run("Close");
	run("Collect Garbage");
	}

Information about your version of Java:

  os.arch => amd64
  os.name => Windows 7
  os.version => 6.1
  java.version => 1.6.0_24
  java.vendor => Sun Microsystems Inc.
  java.runtime.name => Java(TM) SE Runtime Environment
  java.runtime.version => 1.6.0_24-b07
  java.vm.name => Java HotSpot(TM) 64-Bit Server VM
  java.vm.version => 19.1-b02
  java.vm.vendor => Sun Microsystems Inc.
  java.vm.info => mixed mode
  java.awt.graphicsenv => sun.awt.Win32GraphicsEnvironment
  java.specification.name => Java Platform API Specification
  java.specification.version => 1.6
  sun.cpu.endian => little
  sun.desktop => windows
  file.separator => \

The up-to-date check says: UP_TO_DATE

Information relevant to JAVA_HOME related problems:

  JAVA_HOME is set to: C:\FIJI\Fiji.app/java/win64/jdk1.6.0_24//jre
  imagej.dir => C:\FIJI\Fiji.app

Information about the version of each plugin:

Activated update sites:
ImageJ: http://update.imagej.net/ (last check:20140623212553)
Fiji: http://fiji.sc/update/ (last check:20140623232555)

Files not up-to-date:
  035497e9 (LOCAL_ONLY) 20140625191733 macros/Timelapse1.ijm

Comment 1 Wayne Rasband 2014-06-25 13:45:34 CDT

The macro is opening at least three images but only closing two. Try replacing

   run("Close");
   run("Close");
   run("Collect Garbage");

with

   run("Close All");

Comment 2 Curtis Rueden 2014-06-25 13:48:38 CDT

Angus, please give Wayne's suggestion a try and let us know how it goes.

Comment 3 amc910 2014-06-25 14:08:46 CDT

I've changed the macro accordingly and am running it again now. Looking at Task Manager the ImageJ process memory is still creeping up as the macro loops.
The memory monitor on Fiji itself also seems to be increasing, when it pops up on the status bar.

It will take a little while (30 minutes or so) to verify that this memory increase will eventually halt the macro again, but it looks like it's doing the same things as before.

Cheers,
Angus

Comment 4 Curtis Rueden 2014-06-25 14:25:47 CDT

Note that the Task Manager will not accurately report how much memory ImageJ is actually using at the moment. See this FAQ entry on the ImageJ wiki:

 http://wiki.imagej.net/Frequently_Asked_Questions#Why_does_ImageJ_not_release_any_memory_back_to_the_system.3F

But ImageJ's "Monitor Memory" command is good to watch. If it continually increases without dropping, even when you click on the status bar (which triggers a garbage collection), that may indicate a memory issue.

Comment 5 amc910 2014-06-25 15:00:24 CDT

To update, Fiji has indeed run out of memory, and the macro has been aborted.
I tried clicking the status bar while the macro was running, as well as activating a Cleanup Garbage command through the menu system, but neither seemed to do anything.

Ang

Comment 6 Johannes Schindelin 2014-06-25 16:18:09 CDT

There are a couple of things you can do to diagnose what keeps those references to allocated memory... You could give JVisualVM a try, see http://fiji.sc/Debugging#Debugging_memory_leaks

Comment 7 Wayne Rasband 2014-06-25 21:31:35 CDT

This appears to be SCIFIO bug. After running the following macro with "Use SCIFIO when opening files" checked in Edit>Options>ImageJ2 there is 1218MB allocated, which cannot be reclaimed by clicking in the status bar or in the "Memory" window. After running the macro with "Use SCIFIO when opening files" unchecked there is 12MB allocated after memory is reclaimed. I also noticed that the macro runs 6 times slower when "Use SCIFIO when opening files" is checked.

  setBatchMode(true);
  newImage("Untitled", "8-bit ramp", 1024, 1024, 1);
  dir = getDirectory("temp");
  saveAs("tif", dir+"temp.tif");
  close;
  for(image=1; image<=10; image++) {
     for(drop=1; drop<=24; drop++)
        open(dir+"temp.tif");
     run("Images to Stack", "name=Filler title=[] use");
     run("Make Montage...", "columns=5 rows=5 scale=1 first=1 last=24 increment=1 border=0 font=12");
     saveAs("tif", dir+"temp2.tif");
     run("Close All");
  }
  ok = File.delete(dir+"temp.tif")
  ok = File.delete(dir+"temp2.tif")

Comment 8 amc910 2014-06-26 03:45:24 CDT

Morning everyone,

I've disabled SCIFIO and have run the macro again, and all issues seem to have been dealt with.
Memory is correctly being re-used at the end of each loop.
In addition, the time for a loop to run has reduced dramatically, from ~2 minutes per montage to ~10 seconds! (this was all in the time taken to open up the images before converting to a stack. Both the stack and montage commands were very quick).

I consider this issue solved on my end. I hope letting you know about this bug was useful.

Thank you everyone,
Angus

Comment 9 Mark Hiner 2014-07-22 08:17:52 CDT

Sorry for not linking this earlier, but there is a confirmed bug on the SCIFIO/ImageJ2 side, specifically with the legacy layer:

https://github.com/imagej/imagej-legacy/issues/67

This is leading to undesirable hard references to images that do not get cleaned up when an ImagePlus is closed... so there is a true memory leak.

Comment 10 Mark Hiner 2014-07-22 08:21:19 CDT

*** Bug 863 has been marked as a duplicate of this bug. ***

Comment 11 Mark Hiner 2014-08-07 10:36:28 CDT

*** Bug 884 has been marked as a duplicate of this bug. ***

Comment 12 Mark Hiner 2014-08-08 14:55:08 CDT

This should be fixed in the latest Fiji (imagej 2.0.0-rc-10)