NOTICE! This is a static HTML version of a legacy Fiji BugZilla bug.

The Fiji project now uses GitHub Issues for issue tracking.

Please file all new issues there.

Bug 630 - running QuickPALM gives an error
running QuickPALM gives an error
Status: RESOLVED FIXED
Product: Fiji
Classification: Unclassified
Component: Plugins
unspecified
Other Linux
: P5 enhancement
Assigned To: ImageJ Bugs Mailing List
Depends on:
Blocks:
 
Reported: 2013-07-11 07:36 CDT by Olex
Modified: 2013-07-13 14:15 CDT
1 user (show)

See Also:


Attachments
A very small subset of the example image, just enough to reproduce the problem (1.01 MB, image/tiff)
2013-07-13 11:55 CDT, Johannes Schindelin

Description Olex 2013-07-11 07:36:18 CDT
I am using ImageJ/fiji on the cluster through the command line (in a headless mode - without GUI).

I installed QuickPalm plug-in manually by typing: 
./Build.sh plugins/QuickPALM_.jar

When I try to use plugin in the script (example: run("Analyse Particles", "minimum=2 maximum=4 image=127..";), it gives me an error: 

java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at ij.Command.runPlugIn(Command.java:148)
        at ij.Command.runCommand(Command.java:97)
        at ij.Executer.run(Executer.java:64)
        at ij.IJ.run(IJ.java:260)
        at ij.macro.Functions.doRun(Functions.java:589)
        at ij.macro.Functions.doFunction(Functions.java:89)
        at ij.macro.Interpreter.doStatement(Interpreter.java:219)
        at ij.macro.Interpreter.doBlock(Interpreter.java:553)
        at ij.macro.Interpreter.doStatement(Interpreter.java:255)
        at ij.macro.Interpreter.doFor(Interpreter.java:499)
        at ij.macro.Interpreter.doStatement(Interpreter.java:237)
        at ij.macro.Interpreter.doStatements(Interpreter.java:207)
        at ij.macro.Interpreter.run(Interpreter.java:104)
        at ij.macro.Interpreter.run(Interpreter.java:74)
        at ij.macro.Interpreter.run(Interpreter.java:85)
        at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:123)
        at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:107)
        at ij.IJ.runMacroFile(IJ.java:127)
        at ij.ImageJ.main(ImageJ.java:683)
        at fiji.Main.main(Main.java:130)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at imagej.ClassLauncher.launch(ClassLauncher.java:224)
        at imagej.ClassLauncher.run(ClassLauncher.java:161)
        at imagej.ClassLauncher.main(ClassLauncher.java:72)
Caused by: java.awt.HeadlessException
        at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:159)
        at java.awt.Window.<init>(Window.java:432)
        at java.awt.Frame.<init>(Frame.java:403)
        at java.awt.Frame.<init>(Frame.java:368)
        at ij.io.SaveDialog.save(SaveDialog.java:183)
        at ij.io.SaveDialog.<init>(SaveDialog.java:35)
        at QuickPALM.ParticleSaver.setup(ParticleSaver.java:35)
        at QuickPALM.MyDialogs.analyseParticles(MyDialogs.java:392)
        at QuickPALM.Analyse_Particles.run(Analyse_Particles.java:29)
        at ij.IJ.runUserPlugIn(IJ.java:194)
        at ij.IJ.runPlugIn(IJ.java:160)
        ... 31 more

Does this mean that QuickPALM does not support headless mode or I did not install it properly?

P.S.: In my script I do specify that I am running macros in headless mode with the build-in function: setBatchMode(true).
Comment 1 Johannes Schindelin 2013-07-11 09:44:10 CDT
The culprit can be seen here: http://fiji.sc/ParticleSaver.java:35

The SaveDialog is an AWT dialog and cannot be instantiated in headless mode. We have a work-around for the GenericDialog (which is also an AWT dialog) that could be ported to the SaveDialog (and while at it, to the OpenDialog, too).

For that to work, you would need to make sure that a "file" is specified in your run() call.

Please note that I am just finishing up the migration of the headless support to ImageJ2. If you are up for some testing (which would be very much appreciated), I could work on the SaveDialog support even this week, otherwise it will have to wait a little, unfortunately.
Comment 2 Olex 2013-07-11 09:55:38 CDT
I would like to help! 

Just to make sure that I understood you correctly, would you like me to test ImageJ2 in a headless mode?
Comment 3 Olex 2013-07-11 10:01:54 CDT
I would like to help! 

Just to make sure that I understood you correctly, would you like me to test ImageJ2 in a headless mode?
Comment 4 Johannes Schindelin 2013-07-11 10:47:41 CDT
Actually, it would still be Fiji in headless mode.

Remember: Fiji Is Just ImageJ ;-) Fiji acquired many a function over time that was required by work in life sciences. Some of those functions are not exactly specific to life sciences, so we jumped on the band wagon of ImageJ2 (whose mission is to make a next-generation ImageJ that does everything ImageJ 1.x did, but with a more flexible architecture and a more scalable development process).

That means that parts of Fiji, such as the Updater, moved into ImageJ2 (so the "Update Fiji" command actually just hands off to the ImageJ2 updater). Fiji benefits from this in two ways: the updater is now developed in a robust style (with automatic regression testing, based on well-tested core components such as scijava-common and ij-core), and Fiji can focus more on life science.

So you are lucky in that I am in the middle of moving all that headless stuff into ImageJ2 (but of course, Fiji still has the headless mode, it will just simply hand off to ImageJ2 to do it in the very near future), so I am already basically working on the parts of the code you need enhanced.

What will this mean for you? I will upload testing packages to a temporary place when I have anything for you to test. It also means that you can steer the development into the direction you need, for mutual benefit.
Comment 5 Olex 2013-07-11 11:13:55 CDT
Sounds good! 

On a different note, when I was trying to run executable ImageJ-linux64 (downloaded from fiji and ImageJ2 websites) from the command line it gave me an error: 

-bash: ./ImageJ-linux64: Permission denied

The executable with the same name (ImageJ-linux64) that was built by fiji does work.
Comment 6 Johannes Schindelin 2013-07-11 13:23:06 CDT
Please file new tickets for new issues (such as 'downloaded ImageJ-linux64 is not executable') because other people will have the same question but no way to find my answer.

The answer is: for security, downloaded files are never executable by default on Linux/MacOSX/Unix. You have to mark them as executable before you can execute them: "chmod a+x ImageJ-linux64".
Comment 7 Johannes Schindelin 2013-07-11 13:24:29 CDT
Regarding the original issue: some debugging later, it appears that this is a curious issue: The SaveDialog should have found that we're running inside a macro and not instantiated any Frame.

Will keep you posted.
Comment 8 Johannes Schindelin 2013-07-11 13:31:40 CDT
The funny thing is: if I run this little macro (because I have no images to test... maybe you want to make it easier for me to reproduce your problem by providing me with a small sample image and a macro that triggers the error on your side?)

-- snip --
run("Blobs (25K)", "");
run("Analyse Particles", "minimum=5 maximum=4 image=106 smart oonline stream file=[/home/gene099/Particles Table.xls] pixel=30 accumulate=0 update=10 _image=imgNNNNNNNNN.tif start=0 in=50 _minimum=0 local=20 _maximum=1000 threads=50");
-- snap --

it shows a completely different error message:

-- snipsnap --
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at ij.Command.runPlugIn(Command.java:148)
	at ij.Command.runCommand(Command.java:97)
	at ij.Executer.run(Executer.java:64)
	at ij.IJ.run(IJ.java:265)
	at ij.IJ.run(IJ.java:316)
	at QuickPALM.ReconstructionViewer.<init>(ReconstructionViewer.java:59)
	at QuickPALM.Analyse_Particles.run(Analyse_Particles.java:65)
	at ij.IJ.runUserPlugIn(IJ.java:195)
	at ij.IJ.runPlugIn(IJ.java:160)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at ij.Command.runPlugIn(Command.java:148)
	at ij.Command.runCommand(Command.java:97)
	at ij.Executer.run(Executer.java:64)
	at ij.IJ.run(IJ.java:265)
	at ij.macro.Functions.doRun(Functions.java:589)
	at ij.macro.Functions.doFunction(Functions.java:89)
	at ij.macro.Interpreter.doStatement(Interpreter.java:219)
	at ij.macro.Interpreter.doStatements(Interpreter.java:207)
	at ij.macro.Interpreter.run(Interpreter.java:104)
	at ij.macro.Interpreter.run(Interpreter.java:74)
	at ij.macro.Interpreter.run(Interpreter.java:85)
	at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:123)
	at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:107)
	at ij.IJ.runMacroFile(IJ.java:127)
	at ij.ImageJ.main(ImageJ.java:683)
	at fiji.Main.main(Main.java:130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at imagej.ClassLauncher.launch(ClassLauncher.java:224)
	at imagej.ClassLauncher.run(ClassLauncher.java:161)
	at imagej.ClassLauncher.main(ClassLauncher.java:72)
Caused by: java.awt.HeadlessException
	at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:173)
	at java.awt.Button.<init>(Button.java:152)
	at ij.plugin.filter.ScaleDialog.makeButtonPanel(ScaleDialog.java:116)
	at ij.plugin.filter.ScaleDialog.run(ScaleDialog.java:63)
	at ij.plugin.filter.PlugInFilterRunner.processOneImage(PlugInFilterRunner.java:262)
	at ij.plugin.filter.PlugInFilterRunner.<init>(PlugInFilterRunner.java:111)
	at ij.IJ.runPlugIn(IJ.java:168)
	... 40 more
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at ij.Command.runPlugIn(Command.java:148)
	at ij.Command.runCommand(Command.java:97)
	at ij.Executer.run(Executer.java:64)
	at ij.IJ.run(IJ.java:265)
	at ij.IJ.run(IJ.java:316)
	at QuickPALM.ReconstructionViewer.<init>(ReconstructionViewer.java:59)
	at QuickPALM.Analyse_Particles.run(Analyse_Particles.java:65)
	at ij.IJ.runUserPlugIn(IJ.java:195)
	at ij.IJ.runPlugIn(IJ.java:160)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at ij.Command.runPlugIn(Command.java:148)
	at ij.Command.runCommand(Command.java:97)
	at ij.Executer.run(Executer.java:64)
	at ij.IJ.run(IJ.java:265)
	at ij.macro.Functions.doRun(Functions.java:589)
	at ij.macro.Functions.doFunction(Functions.java:89)
	at ij.macro.Interpreter.doStatement(Interpreter.java:219)
	at ij.macro.Interpreter.doStatements(Interpreter.java:207)
	at ij.macro.Interpreter.run(Interpreter.java:104)
	at ij.macro.Interpreter.run(Interpreter.java:74)
	at ij.macro.Interpreter.run(Interpreter.java:85)
	at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:123)
	at ij.plugin.Macro_Runner.runMacroFile(Macro_Runner.java:107)
	at ij.IJ.runMacroFile(IJ.java:127)
	at ij.ImageJ.main(ImageJ.java:683)
	at fiji.Main.main(Main.java:130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:616)
	at imagej.ClassLauncher.launch(ClassLauncher.java:224)
	at imagej.ClassLauncher.run(ClassLauncher.java:161)
	at imagej.ClassLauncher.main(ClassLauncher.java:72)
Caused by: java.awt.HeadlessException
	at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:173)
	at java.awt.Button.<init>(Button.java:152)
	at ij.plugin.filter.ScaleDialog.makeButtonPanel(ScaleDialog.java:116)
	at ij.plugin.filter.ScaleDialog.run(ScaleDialog.java:63)
	at ij.plugin.filter.PlugInFilterRunner.processOneImage(PlugInFilterRunner.java:262)
	at ij.plugin.filter.PlugInFilterRunner.<init>(PlugInFilterRunner.java:111)
	at ij.IJ.runPlugIn(IJ.java:168)
	... 40 more
Comment 9 Johannes Schindelin 2013-07-11 14:33:37 CDT
Could you please package a small example image and a macro that together trigger the error and attach it to this issue?
Comment 10 Olex 2013-07-12 09:29:34 CDT
 I was not able to upload them here. I used some alternative method.
Comment 11 Johannes Schindelin 2013-07-12 14:53:15 CDT
At the moment I cannot access websites requiring Javascript. Besides, it would be better to use the builtin Help>Upload Sample Image to upload the sample.

Also: could you provide me with the macro you try to run? Please keep in mind that you can help reduce the time I require to reproduce dramatically (I will still need to spend a couple of hours just to find the problem and to fix it after that).
Comment 12 Johannes Schindelin 2013-07-12 14:55:06 CDT
This is not so much a problem of Fiji, but a security measure unknown to Windows users: files that are downloaded on Linux or MacOSX are never marked as executable. You will have to do that yourself, e.g by using the command-line

    chmod a+x <the-downloaded-file>
Comment 13 Johannes Schindelin 2013-07-12 14:56:36 CDT
Oops please forget comment #12...
Comment 14 Olex 2013-07-13 07:42:46 CDT
There is no Upload Images in Help. Unfortunately, the limit for the attachment here is 5 MB, and my smallest image is 546 MB. Is there any other way to send it to you? The script below is the macros that I used: 

setBatchMode(true);

dir = '/yourDirectoryHere/';
list = getFileList(dir);


Array.print(list);
for (i=0; i<list.length; i++) {
open(dir+list[i]);
run("Split Channels");
close();
selectWindow("C1-"+list[i]);
saveAs("Tiff", dir+"C1-"+list[i]);
run("Analyse Particles", "minimum=2 maximum=4 image=127 smart online pixel=20 accumulate=0 update=100 _image=imgNNNNNNNNN.tif start=0 in=50 _minimum=0 local=20 _maximum=1000 threads=50");
selectWindow("C1-"+(i+1)+" Reconstruction");
saveAs("Tiff", dir+"C1-"+(i+1)+" Reconstruction");
close();
}

I am not sure whether it will be of any use since you do not have my sample...
Comment 15 Johannes Schindelin 2013-07-13 11:55:17 CDT
Created attachment 108
A very small subset of the example image, just enough to reproduce the problem
Comment 16 Johannes Schindelin 2013-07-13 11:56:11 CDT
I am a bit confused... if you open the Help menu, there is no "Upload Sample Image" entry just under "Report a Bug"?

In any case, I finally have access to a browser with Javascript again (when I work on my computer via my phone, I actually only have a text browser without Javascript, and the phone itself cannot run Fiji so the phone's browser is of no use), so I am finally able to access the Google Drive link you sent me.

For the record, you do not have to use original data when you send a macro to reproduce a bug. As I described in http://developer.imagej.net/2011/10/17/art-debugging the most important part about debugging is to make a *small* test case first. I will have to run this procedure a *lot* of times, and the quicker it runs, the quicker I will get at the root of the problem, and the quicker I will be able to come up with a fix (also due to the fact that my time -- as everybody else's -- is limited to around twenty-four hours a day, and a large part of that is not available for work on this particular bug).

So the first thing is to reduce the file size (because loading and processing that size is taking a lot of time).

The first file size reduction is easy: split the channels and only use channel 1 (because that is what you use anyway, uploading any other data than the first channel for the sake of reproducing this bug is unnecessary).

This reduces the file size from 534MB to 178MB.

The next step is also obvious: there was no problem with splitting the channels, so let's skip that completely.

Next, I could downsample the remaining image, but in this particular case it is better to just crop it: as far as QuickPALM is concerned in this case, it is more important to preserve the character of the problem by concentrating on a small section of the image, where all the action is happening.

So cropping out just a small 128x128 part, I reduced the file size to 22MB.

And using "Make Substack..." (tip: if you do not know where in the menu structure a certain command is located, just hit Ctrl+L and type a part of the label) to keep only the first 32 slices reduced the size to a handy 1MB.

This is what I attached as http://fiji.sc/bugzilla/attachment.cgi?id=108.

Further, the macro can be reduced further if all we need to do is to reproduce the problem: no directory scanning is required, no saving, no closing. This is the bare, minimal macro I came up with:

-- snip --
open("small-2.tif");
run("Analyse Particles", "minimum=2 maximum=4 image=127 smart online pixel=20 accumulate=0 update=100 _image=imgNNNNNNNNN.tif start=0 in=50 _minimum=0 local=20 _maximum=1000 threads=50");
-- snap --

Please do not understand this as a criticism, but as encouraging you to reduce the macro and data next time you are reporting a bug; that way, I will not only save time to reduce them myself, but I will be able to work on the issue sooner (because I do not need to download >500MB via a Javascript-enabled browser)...
Comment 17 Johannes Schindelin 2013-07-13 12:26:40 CDT
Progress: I can now start the macro by just adding a " file=[/path/to/file]" that specifies where the results should be saved to. So my minimal macro now looks like this:

-- snip --
open("/tmp/small-2.tif");
run("Analyse Particles", "minimum=2 maximum=4 image=127 smart online pixel=20 accumulate=0 update=100 _image=imgNNNNNNNNN.tif start=0 in=50 _minimum=0 local=20 _maximum=1000 threads=50 file=[/tmp/olex.txt]");
-- snap --

I also fixed a bug in QuickPALM that shows only with small files (see the mail I just sent to Ricardo, you Cc:ed to keep you in the loop).

Now on to figure out why the macro recorder forgot to record the "file" parameter.
Comment 18 Olex 2013-07-13 13:14:09 CDT
Thank you for the advice. I am still learning ImageJ/fiji in order to make all those transformations. 

Please, note each image contains multiple frames.
Comment 19 Johannes Schindelin 2013-07-13 14:15:51 CDT
Yes, I just kept fewer than the original 710 frames because the only thing I needed to reproduce was the error you reported, not the complete procedure. And for that, 32 slices were quite plenty (and very fast to process, I might add).

So here are good and bad news. The bad news are: when I record a macro using the QuickPALM "Analyse Particles", I get something like this:

-- snip --
run("Analyse Particles", "minimum=5 maximum=4 image=106 smart online stream file=[/tmp/Downloads/Particles Table.xls] pixel=30 accumulate=0 update=10 _image=imgNNNNNNNNN.tif start=0 in=50 _minimum=50 local=20 _maximum=1000 threads=50");
-- snap --

Do you notice the "file=[...]" part? This is what was lacking from the macro as presented in comment 14.

The good news is that there is no bug I need to fix; I have no idea whether this was a bug in the ImageJ version you used for recording, but at least in 1.47t it works correctly.