N-Dimensional Support

[andrew]

Will highlighted that the new N-Dimensional model will affect how we can represent ROI, and specifically link them to a particular dimension.

• Different images will have different dimensions
• We may want to have a particular ROI on (x,y) but independent of all others, Softworx for instance draws an ROI on the screen, but measurements change depending of Z,T.
• We could have a situation where we have a different ROI tables, or templates of tables for types of imaging.

Josh, 7th Dec notes from ASCB. "The decision was to start work (simultaneously with the 4.2 release, but well integrated into the iterations!) on an experimental API (planning toward for OME 5). This would include a new combined DataService? (Curtis suggests "OMEIS" ~ :) ) with base methods:

Dimension[] getDimensions(); DataBlob? getSlice(Dimension[] reorderedDims,

long[] start, long[] stop, long[]);

Simpler methods can be built on top of these, and even the legacy APIs can be kept.

One open questions is where to store the metadata (planeinfo, etc.) We might prefer to keep it in an HDF file, for example.

The plan is based on the assumption that we will work toward reading planes directly from the original files with bioformats, which, however, requires re-import functionality (i.e. hashing).

The n-dimensional work in the server code will most likely happen with mercurial as a test bed for whether or not it works for us (and hopefully, to make merging later)"

Use Cases

Sparse Data Collection

• As I understand it, we are *NOT going to support sparse data* with complex pixel types. However it could be supported with Image Container  http://trac.openmicroscopy.org.uk/omero/ticket/2552 (Multiple pixels is another possibility but more work & break more code).
• Sparse Data is actually quite a common requirement (will become more so) E.g.:
  • I have a stack of fluorescent planes (channel 0) but only 1 plane for transmitted light (channel 1)
  • I acquire a DAPI channel and a FLIM channel (which has bins that are not in the DAPI channel).
  • I have a transmitted light plane and a wavelength scan for each time point.
• Subsampling of certain dimensions (e.g. 1ms. bursts of high-resolution over a given period) * This could be handled by having burst_delta_T as a different dimension from T.

Flim

• Each pixel is an n-bin histogram. Do all the pixels have the same number of bins? If not, it seems like sparse data?

• FLIM (phase method - E.g. Lambert). Extra dimensions are Frequency and Phase. 7 Dimensions in total.

• FLIM with polarisation.  http://dx.doi.org/10.1016/j.copbio.2009.01.004 Curtis: "recording whether each event has a positive or negative spin. I don't know all the details of this yet. We have spoken to Klaus Suhling about his work in this area"

Other modalities

• AFM: multiple parameters over scan lines. Needs flexible modelling of the parameters since they are created so frequently. Again, not sure if this is sparse data?
• FCM: multiple parameters per cell
• HCS: representing entire plate in a single structure

EM

• Tomography. Extra dimension is Angle. Usually tilt in 1 direction only, but 2 is possible. "Tilting is a rotation. And rotation is nothing more than one dimension, I agree. But the unit of this dimension is again a new one. It is not nanometers, it is not seconds. it is angles. And you could tilt around any axis. The axis of rotation has one direction and a center of rotation, all with respect to a reference image (not tilted). AT the very least you should be able to store these informations with the data. And the increment is often not constant. There are schemes where you try to have the increment in the sinus of the angle constant. Jean-Marc 13.1.2010"

• Spectral data: "In EELS you could have a full spectum of energies for each pixel. Likewise in EDX but with a resolution different from the EM data. Johannes 12.1.2010"

Other issues

• Tiling, or "chunking", of very large dimensions. Related to / same as stitching representation (is there a ticket for this?)
• Hashing (corruption detection) and/or compression of various/arbitrary/specific dimensions

Proposals

This proposal has been updated in line with the 29th June discussion to produce proposal version 4.

The current proposal included here can be edited at Resources/112-NDim/Proposal.

<!--

Proposal Version 4 (based on Will's Proposal)

• Define the dimensions using a controlled vocabulary.
• Have a single indexes attribute that has comma separated list.
• Define both Incremental and Absolute dimensions.
• Specifying units for all dimensions (including X, Y, Z, T) allows us to use non-standard units. E.g. Angstroms or nm for EM pixels or nano-seconds for FLIM time-bins etc. One way of ensuring we all use the same standard (e.g. "Seconds" vv "seconds" vv "s" or "um" vv "microns") is to use an ontology. See units ontology  http://obo.cvs.sourceforge.net/viewvc/obo/obo/ontology/phenotype/unit.obo

Just ignore the next XML block

-->
<example xmlns:OME="http://www.openmicroscopy.org/Schemas/OME/2010-06"
	xmlns:BIN="http://www.openmicroscopy.org/Schemas/BinaryFile/2010-06"
	xmlns:xsd="http://www.w3.org/2001/XMLSchema">
	<xsd:import namespace="http://www.openmicroscopy.org/Schemas/OME/2010-06"
		schemaLocation="
		http://cvs.openmicroscopy.org.uk/svn/specification/Xml/Working/ome.xsd"/>
	<!--

Updated Example XML. Proposed Dimension elements under Pixels

-->
	<IncrementingDimension name="ExcitationLamba" unit="nm" Size="20" Key="L" Increment="10" StartValue="405">
		<OME:Description>Excitation Wavelength Window of 15 nm</OME:Description>
	</IncrementingDimension>
	<IncrementingDimension name="X" unit="um" Size="3" Key="X" Increment="0.06631"/>
	<AbsoluteDimension name="Angle" unit="degrees" Size="3" Key="A">
		<OME:Description>The angle of acquisition</OME:Description>
	</AbsoluteDimension>
	<!--

On Pixels the DimensionOrder attribute now is an array of the keys

	-->
	<OME:Pixels DimensionOrder="X,Y,Z,L,A" ID="Pixels:1" Type="int16">
		<!-- List of dimensions used -->
	</OME:Pixels>
	<!--

On Plane add an indices array that replaces TheT, TheC, etc.

	-->
	<Plane Indices="0,0,0,0,0">
		<AbsoluteValue key="T">0.0</AbsoluteValue>
		<AbsoluteValue key="A">4e+007</AbsoluteValue>
	</Plane>
	<!--

• Phase-domain FLIM (7 dimensions)
  • Don't have to specify 'Increment' for incrementing dimension (E.g. no acquisition data - see T below or C)
  • If no 'StartValue?' is specified for incrementing dimension, assume 0.

    	-->
    	<OME:Image ID="Image:1" Name="example_R3D_D3D.dv">
    		<OME:Pixels DimensionOrder="X,Y,Z,C,T,P,F" ID="Pixels:1" Type="int16">
    			<IncrementingDimension name="X" size="480" key="X" Increment="0.06631"/>
    			<IncrementingDimension name="Y" size="480" key="Y" Increment="0.06631"/>
    			<IncrementingDimension name="Z" size="1" key="Z" Increment="0.2"/>
    			<IncrementingDimension name="T" size="1" key="T"/>
    			<IncrementingDimension name="C" size="1" key="C"/>
    			<AbsoluteDimension name="Phase" unit="degree" size="12" key="P"/>
    			<IncrementingDimension name="Frequency" unit="Hz" size="1" key="F" Increment="30"/>
    			<OME:Channel EmissionWavelength="457" ExcitationWavelength="360" ID="Channel:1:0"/>
    			<BIN:BinData BigEndian="false" Length="0"/>
    			<OME:Plane Indices="0,0,0,0,0,0,0">
    				<AbsoluteValue key="P">0</AbsoluteValue>
    			</OME:Plane>
    			<OME:Plane Indices="0,0,0,0,0,1,0">
    				<AbsoluteValue key="P">30</AbsoluteValue>
    			</OME:Plane>
    			<OME:Plane Indices="0,0,0,0,0,2,0">
    				<AbsoluteValue key="P">60</AbsoluteValue>
    			</OME:Plane>
    			<OME:Plane Indices="0,0,0,0,0,3,0">
    				<AbsoluteValue key="P">90</AbsoluteValue>
    			</OME:Plane>
    			<OME:Plane Indices="0,0,0,0,0,4,0">
    				<AbsoluteValue key="P">120</AbsoluteValue>
    			</OME:Plane>
    			<!--
    			... other planes here ...
    			-->
    		</OME:Pixels>
    	</OME:Image>
    	<!--
    

• SPIM (Tilt Angle)

  	-->
  	<OME:Image ID="Image:1" Name="example_R3D_D3D.dv">
  		<OME:Pixels DimensionOrder="X,Y,Z,C,T,A" ID="Pixels:1" Type="int16">
  			<IncrementingDimension name="X" size="480" key="X" Increment="0.06631" unit="um"/>
  			<IncrementingDimension name="Y" size="480" key="Y" Increment="0.06631" unit="um"/>
  			<IncrementingDimension name="Z" size="1" key="Z" Increment="0.2" unit="um"/>
  			<IncrementingDimension name="T" size="100" key="T" Increment="30" unit="seconds"/>
  			<IncrementingDimension name="C" size="1" key="C"/>
  			<IncrementingDimension name="Angle" unit="degree" size="4" key="A" Increment="45"/>
  			<OME:Channel EmissionWavelength="528" ExcitationWavelength="490" ID="Channel:0:0"/>
  			<!-- Don't need planes since dimension increment is stated above and
  				no precise values are needed for individual planes
  				Also must assume Increment has default of 1
  			-->
  			<BIN:BinData BigEndian="false" Length="0"/>
  		</OME:Pixels>
  	</OME:Image>
  	<!--
  

• Big Lamda (excitation sweep for each emission wavelength window/channel)

  	-->
  	<Instrument ID="Instrument:0">
  		<Filter ID="Filter:0:0">
  			<TransmittanceRange CutIn="400" CutOut="450"/>
  		</Filter>
  		<Filter ID="Filter:0:1">
  			<TransmittanceRange CutIn="450" CutOut="500"/>
  		</Filter>
  		<Filter ID="Filter:0:2">
  			<TransmittanceRange CutIn="500" CutOut="550"/>
  		</Filter>
  		<Filter ID="Filter:0:3">
  			<TransmittanceRange CutIn="550" CutOut="650"/>
  		</Filter>
  	</Instrument>
  	<Image ID="Image:1" Name="bigLamda.dv">
  		<Pixels DimensionOrder="X,Y,Z,C,T,Lamda" ID="Pixels:1" Type="int16">
  			<IncrementingDimension name="X" size="480" key="X" Increment="0.06631" unit="um"/>
  			<IncrementingDimension name="Y" size="480" key="Y" Increment="0.06631" unit="um"/>
  			<IncrementingDimension name="Z" size="1" key="Z" Increment="0.2" unit="um"/>
  			<IncrementingDimension name="T" size="100" key="T" Increment="30" unit="seconds"/>
  			<IncrementingDimension name="C" size="4" key="C"/>
  			<IncrementingDimension name="ExcitationLamba" unit="um" size="20" key="Lamda" Increment="10">
  				<OME:Description>Excitation Wavelength Window of 15 nm</OME:Description>
  			</IncrementingDimension>
  			<Channel ID="Channel:0:0">
  				<LightPath>
  					<EmissionFilterRef ID="Filter:0:0"/>
  				</LightPath>
  			</Channel>
  			<Channel ID="Channel:0:1">
  				<LightPath>
  					<EmissionFilterRef ID="Filter:0:1"/>
  				</LightPath>
  			</Channel>
  			<Channel ID="Channel:0:2">
  				<LightPath>
  					<EmissionFilterRef ID="Filter:0:2"/>
  				</LightPath>
  			</Channel>
  			<Channel ID="Channel:0:3">
  				<LightPath>
  					<EmissionFilterRef ID="Filter:0:3"/>
  				</LightPath>
  			</Channel>
  			<!-- Don't need planes since dimension increment is stated above and
  								no precise values are needed for individual planes -->
  			<BIN:BinData BigEndian="false" Length="0"/>
  		</Pixels>
  	</Image>
  	<!--
  

-->

</example>

[jmoore]

Notes from ASCB:

** #1 Sun 12-17 07:26 ([[OmeWork#174]])
...
 - NDIM
  -- conflating pixel types and dimension order: what we support (server) or what's in file (bioformats)
  -- wider ranges (diversity); one for in file/one for what we're giving you
  -- bioformats has a clear way forward
  -- keeping performance in mind up-front as opposed to do just focusing on API
  -- cF. fs ~ reading bytes on the fly?
  -- keeping a simple planar API?
  -- bioformats.openData() ~ assumed planar, but could be block based
  -- enumerating the types? tough
  -- reading the data v. display
  -- what can't be represented as a slider?
  -- sparseness!!! (should be in introspection of dimensions?)
   --- merging; turning them into two image scenario?
  -- Lifetime metadata in HDF as well? (PlaneInfo?)
  -- dimensional metdata:
   --- continugous?
   --- measurement?
   --- scale? units?
   --- "usually a slider?"
   --- sparseness?
   --- defaults? use index 0, "bin the histogram", ... trying not to crash
  -- DataService(instead of ImageService)
   --- Using P/W as another dimension
  -- @Start testing NDIM before 4.2
   --- Use mercurial to make merging easier
   --- Only within iteration system

[jmoore]

.#1 Modelling NDIM & SPW 16:03 ([[OmeWork#199]])

 - NDIM: http://www.ome-xml.org/ticket/112
  -- previously
   --- FLIM --> "we need a sixth dimension"
  -- HDF
   --- how is it encoded? can reflect upon the dimensions
   --- will have to add some metadata on what each dimension is
   --- is it fast? fastest we know of
   --- cF. for last TIFF section, will have to read, seek, read, seek
  -- BioFormats
   --- 2D, other are optional? (implies changing everything)
   --- an issue of interpretation. currently, we are perhaps too rigid.
   --- in medical imaging, axial representation
   --- Will: discuss going back to XYZ and then have additional
   --- Chris: there's a reason that everyone discuss spatiotemporal
  -- Chris: 2 problems - storage & display problem
   --- storage = spacetime with measurements
   --- display
    --- xyz aren't really the same dimensions, not the same scale
   --- normalizing for axial, e.g.
  -- Jason: Trying to move this forward
   --- There will always be a field which doesn't care about some dimension
   --- We already have a purchase with our current structure because our field is 5D.
   --- Currently folks are writing out OME-TIFF since it is understood
   --- Anyway to provide what we have and also move forward?
   --- Donald: not that difficult. "traditional" (OME5D) and "ndim"
    ---- Will: delegating from our existing methods to the new NDIM methods
    ---- Chris: problem is at XML/file-exchange level
    ---- Jason: file-exchange is largely solved. cF. EM
    ---- Chris: hybrid approach makes it easy to hide behind API
    ---- Chris: but as a OME-implementor I have to support two file formats
     ----- 2 readers, 2 writers, ...
     ----- Donald: we still haven't solved all the technical issues (cF. ROI)
     ----- Jason: concrete example of a NDIM Roi that is beyond draw a circle?
      ------ Volume over time?
      ------ ROI with the height of the AFM?
  -- Josh: Is this the difference between OME-TIFF and OME-HDF (something new and experimental)?
   --- Chris: for us, it's easy because we have an API
   --- but we don't ship a library.
   --- Do we provide a reference-implementation library for doing everything/conversions/etc.
   --- e.g. AFM and FLIM. EM?
   --- no one really cares about the XML
   --- Donald: allows us to upgrade things, etc.
   --- Will: we'll have the code anyway
  -- summarize
   --- have to address what are the minimum: xy(zt)
   --- @Andrew: find a list of the places in the model which are affected
   --- is there a resounding chorus of "this structure sucks"?
    ---- we store images
   --- (not hurting our momentum)
   --- determine short-term solutions for the likes of EM
    ---- Re-using Z for angle?
    ---- similar to FLIM solution.

[andrew]

I went through the element in the schemas and asked two questions.

Does this refer directly to the dimensions of our 5-D data, e.g. "theZ"? Does this store information that relates to a specific dimension, e.g. "PositionX"?

y is yes to first, ? is yes to second, n is no to both

Element Name    	5-D dim	X	Y	Z	T	C	Comments
Pixels          	yes    	y	y	y	y	y	
Plane           	yes    	?	?	?y	y	y	xyz stage position
TiffData        	yes    	n	n	y	y	y	
StageLabel      	no     	?	?	?	n	n	xyz position
OTF             	no     	?	?	n	n	n	x and y size
Plate           	no     	?	?	n	n	n	well origin
PlateAcquisition	no     	n	n	n	?	n	start/end time
WellSample      	no     	?	?	n	?	n	xy position, timepoint
ROI             	yes    	n	n	y	y	y	
Line            	no     	?	?	n	n	n	2d position
Rectangle       	no     	?	?	n	n	n	2d position
Mask            	no     	?	?	n	n	n	2d position
Ellipse         	no     	?	?	n	n	n	2d position
Point           	no     	?	?	n	n	n	2d position
Polyline        	no     	?	?	n	n	n	2d position
Path            	no     	?	?	n	n	n	2d position
Label           	no     	?	?	n	n	n	2d position

Hope this makes sense

[jmoore]

.#1 Model mtg. 16:10 ([[OmeWork#205]])

 - NDIM
  -- important for ROI & FLIM
  -- FLIM has workaround, but not perfect
  -- http://www.ome-xml.org/ticket/112
  -- https://trac.openmicroscopy.org.uk/omero/wiki/WorkPlan/NDimensionalData
  -- Andrew: shoe-horning data into 5D
  -- 3 things need to change:
   --- API access: NDIM only used by necessity
   --- how do we store the data?
   --- how to link to specific "planes"? (BIG UNKNOWN)
    ---- not visible via the graphs
    ---- FLIM: planeinfo is a time thing, attached to channel
    ---- each bin of histogram
    ---- Josh: PlaneInfo type hierarchy?
    ---- Chris: Aren't we talking about ROIs?
    ---- nested set of vectors across any dimensions
  -- split PlaneInfo
   --- Dimension
   --- DimensionData
   --- TraversalInstructions
  -- what are your slices? ("axis of work")
  -- Jean-Marie and Donald prepare something for discussion

[andrew]

There are notes from the trip to Imperial available on request from Andrew or Scott.

[jmoore]

Modelling meeting, incl. Jean-Marie, Will, Donald, Chris, Josh

 - NDIM
  -- Clarifying the use of the two arrays (notes from Paris)
  -- Tricky: angles
  -- Will: At least xyzt? Yes.
  -- Jean-Marie: where do we attach metadata?
  -- Jean-Marie: need units!
  -- PlaneInfo as a set of arrays...
  -- Chris: re. translation do we want to encode information rather than objects
  -- Need to have enough information for representing the dimension
  -- Storing Deltas?
  -- A place to add documentation to the dimension
  -- Remove all specialized attributes and increment concepts
  -- Possibly keeping xyzt (and make them optional)
  -- Donald: just add "value" to each as opposed to start/increment
  -- Chris: possibly add start/increment/end to prevent searching for stride
  -- Donald: something like "this value is not incremented" boolean
  -- Chris: types for the dimension ("IncrementingDim", "AbsoluteDim",...)
  -- Conclusion:
   --- array solution
   --- will initially attempt single array (possibly two)
   --- need to decide how to represent dimensions and planeinfo
   --- next step: modify the examples (on separate changes?!)
   --- including how filters are linked etc.
   --- and what the db then looks like.
   --- (this is the first modelling effort to take place)
   --- Andrew: possibly avoid "feature" name? (it's a reuse)
  -- Jean-Marie: Still need to discuss sparseness
   --- J-M: Will sometimes need to link two images into one
   --- Only want one thumbnail
   --- Do we have to use 2 rendering engines
   --- Makes working with the data very hard
   --- Nothing prevents us from using Pixels (but that still 2 entities)
   --- Using pixels for sparseness (lots of good examples) but gets polluted with decon, etc.
   --- Need to describe the mapping better, to know what the image actually is.
   --- So would need stitching as well for other things
   --- or maybe we're making things too complicated and just need a simple link table
   --- Requires moving a lot down to pixels, and we don't like the name "Pixels" anyway
   --- Not many (if any) formats support sparseness
   --- Donald: FM is represented that way (2 instruments at the same time)
   --- Andrew: using SPW for stitching? possibly.
   --- To decide: new type or pixel renaming (&pushing things down)

[andrew]

Proposal Version 1

Discussion in Dundee (08/06/2010), Will, Donald, Chris, J-M

• Description the spatial location: Use of X, Y, Z, T (as current) and an optional Theta (Theta_1, Theta_2, Theta_3) as indexes, with unified unit e.g. degrees.
• C becomes Feature and acquisition metadata should be linked to it.

Usage

What needs to change at the model level?

• DimensionOrder: Replace with indexes where X=0 Y=1 Z=2 T=3 Theta[]=4,5,6 Features[] indexes start at 7.
• PlaneInfo:
  • TheC becomes TheF
  • add TheTheta[]
• Pixels:
  • SizeC becomes SizeF[]
  • add SizeTheta[], ThetaIncrement[]
  • remove WaveStart, WaveIncrement
• Shape
  • TheC becomes TheF
  • add TheTheta[]
• Feature should be defined in NameSpace, we will provide a list of predefined terms.

What needs to change at the omero level?

• RenderingDef:
  • add defaultTheta[]
• ChannelBinding will be modified.

Back to the use cases above?

• Covered:
  • Current microscopy data
  • FLIM
  • AFM
  • SPIM
  • arbitrary feature data e.g. analysis from CellProfiler
• Not covered:
  • Subsampling will not be covered, this should probably be handled by image-image link.

[andrew]

Proposal Version 2

Discussion in Dundee (09/06/2010), Will, Donald, Chris, J-M, Andrew

Adjusted Version 1

• Description the spatial location: Use of X, Y, Z, T (as current) and an optional Manipulation/Angle/Rotation (name TDB) as index.
• C becomes Feature and acquisition metadata should be linked to it.

Usage

What needs to change at the model level?

• DimensionOrder: Use symbols where X, Y, Z, T, M/A/R and Features[] String[] such that for example:

setDimensionOrder(new char[] { 'X', 'Y', 'M', 'Z', T'}, new String[] {'flim'})...

• PlaneInfo:
  • TheC becomes TheF
  • add Theta_1, Theta_2, Theta_3, etc. attributes for spacial manipulation
  • add TheManipulation
• Pixels:
  • SizeC becomes SizeF[]
  • add SizeManipulation[]
  • remove WaveStart, WaveIncrement
• Shape
  • TheC becomes TheF
  • add TheTheta[]
• Feature should be defined in NameSpace, we will provide a list of predefined terms.

What needs to change at the omero level?

• RenderingDef:
  • add defaultManipulation
• ChannelBinding will be modified.

Back to the use cases above?

• Covered all of those in Version 1

[andrew]

Proposal move out to new wiki page. The wiki page is included into the ticket description.

see: Resources/112-NDim/Proposal