ChipMaker2
==========
:stem: latexmath
:source-highlighter: highlight.js

Keywords: chips, machine learning

== Summary
This is an advanced demonstration of using the *ChipMaker* combo plugin.
In order to facilitate robust training of machine learning algorithms,
we want a large number of images featuring the content of interest.
These image "chips" should span a wide range of acquisition parameters.
This plugin was created to streamline the generation of image chips
with "labels" by automating the sampling of a user-defined parameter
space in a single simulation rather than relying on an infrastructure
of external scripts to iterate through the parameter space.

IMPORTANT: This is a DIRSIG5 specific feature and cannot be used in DIRSIG4.

== Details
In order to facilitate robust training, we want to generate a large number
of image chips across a wide range of acquisition parameters. Those include:

* Different targets and/or variants of targets
* Different backgrounds (the context within which the target appears)
* Different ground sampling distances (GSDs)
* Different sensor view angles (zenith and azimuth)
* Different illumination angles (zenith and azimuth)

Historically this has been accomplished using external scripting with a
conventional DIRSIG simulation.  The primary goal of this plugin is to
make it easy to configure all the degrees of freedom in one location
and have the plugin manage the creation of the images.

== Important Files
This section highlights key files important to the simulation.

=== Scene
The scene was developed specifically for this demo.  Although the scene
is very simple, it illustrates techniques that can be used to develop
other scenes appropriate for use in this algorithm training applications.
The goal of the scene construction approach at this time is to create a
large number of contexts for each target of interest. In this case, the
context primarily refers to the background. To create a robust training set
for a given target, we want images of that target against a large number
of backgrounds.  There are currently two ways to accomplish this with 
the current DIRSIG workflow:

* Create a scene with a divergse background and create a large number of 
  instances of the target within that scene. Each instance of the target
  will be in a unique location and, hence, have a unique background.  In
  this demo, the helicopters use this approach.
* Create a scene with a diverse background and use motion of a small set
  of target instances that are in motion. When the targets are imaged at
  different times, there will be a unique background around the object.
  In this demo, the bomber plane uses this approach.

==== The Background
The background for the scene is defined as a flat terrain with a material
(class) map defining asphalt, concrete, grass and dirt with texture maps.
This object is defined in a bundle (see `geometry/bundles/background`).

==== The KA-27 Helicopters
The KA-27 helicopter is defined in a set of bundles.  There are 6 total
variants of the KA-27: 3 materials variants (Indian, Korean and Russian)
and 2 state variants (idle and active). The material variants use a
unique set of materials for each vehicle (different reflectance maps
featuring nation specific paint schemes and markings). The 2 state variants
define if the rotors on the vehicles are in motion or not.  This results
in 6 permutations of the 3 materials and 2 states.  These are captured 
in the 6 GLIST files within the KA-27 bundle folder:

* `geometry/bundles/ka27/ka27_indian_active.glist`
* `geometry/bundles/ka27/ka27_indian_idle.glist`
* `geometry/bundles/ka27/ka27_korean_active.glist`
* `geometry/bundles/ka27/ka27_korean_idle.glist`
* `geometry/bundles/ka27/ka27_russian_active.glist`
* `geometry/bundles/ka27/ka27_russian_idle.glist`

In order to select these objects in the ChipMaker, the target objects
have the tags defined at the top-level geometry list in the bundle GLIST
files. The example below is from the `ka27_russian_idle.glist` file showing
the tags defined:

[source,xml]
----
include::geometry/bundles/ka27/ka27_russian_idle.glist[]
----


==== The "Badger" Bomber Plane
The "Badger" bomber is defined in a bundle (see `geometry/bundles/badger`).
There is only a single instance of the bomber in the scene, which is 
defined in the `geometry/planes.glist` file. 

In order to select these objects in the ChipMaker, the target objects
have the tags defined at the top-level geometry list in the bundle GLIST
files. The example below is from the `badger.glist` file showing the tags
defined:

[source,xml]
----
include::geometry/bundles/badger/badger.glist[]
----

Note that this bundled object also has a set of 3D "keypoints"
defined, which correspond to the 3D points for the nose, wing tips
and tail in the OBJ file.  These 3D points will be reported in the
JSON "meta data" files with the corresponding 2D coordinates in the
image.

The badger plane is instanced into the scene in two places via the
`geometry/planes.glist` file.  There is static instance for a parked
instance of the plane and a second instance that is is dynamic,
using the DeltaMotion model to define a take-off of the plane over
the runway in the scene.  The total takeoff sequence described in
the motion file (see `geometry/motion/PlaneTakeOff.mov`) spans from
0 - 150 seconds. This time window will be used to capture the plane
at various states in the take off sequence.  Note that we have added
the tag "flying" to the dynamic instance. This tag will be added
to the tags for this instance of the badger.

[source,xml]
----
include::geometry/planes.glist[]
----

.A rendering of the complete scene.
image::images/demo.png[width=512]

The key feature of this scene is that it utilizes the "tagging"
system introduced for DIRSIG5 to add shared and unique "labels" to each
of the 21 objects in the scene.  These tags will be used in the simulation
to select which objects will be sampled in the output image set.

==== Geometry Tags
A more detailed explanation of the tagging system and how tags are
inherited is described in the ChipMaker1 demo.  Below is a list of
the tags in the scene and what targets they map to:

`aircraft`::
   Maps to all aircraft (helicopters and planes) in the scene.
`choppers`::
   Maps to all the helicopters in the scene.
`planes`::
   Maps to all the planes in the scene.
`ka27`::
   Maps to all the KA-27 helicopters in the scene.
`badger`::
   Maps to all the "Badger" planes in the scene.
`indian`::
   Maps to all the Indian aircraft in the scene (in this case, the idle
   and active KA-27 helicopters).
`korean`::
   Maps to all the Korean aircraft in the scene (in this case, the idle
   and active KA-27 helicopters).
`russian`::
   Maps to all the Russian aircraft in the scene (in this case, the idle
   and active KA-27 helicopters).
`idle`::
   Maps to all the idle aircraft in the scene (in this case, the idle
   KA-27 helicopters).
`active`::
   Maps to all the active aircraft in the scene (in this case, the 
   active KA-27 helicopters and the Badger).
`flying`::
   Maps to all the active aircraft in the scene that are airborne (in this
   case, the only the Badger is ever airborne).


== Simulations and Results ==
=== Input Simulation Files
The ChipMaker plugin is configured via a DIRSIG
http://www.dirsig.org/docs/new/jsim.html[JSIM] file.
Refer to the http://www.dirsig.org/docs/new/chipmaker_plugin.html[plugin manual]
for specific details about the format and options.  This demo includes
setups to separately generate helicopter image chips (see
`chopper_chips.jsim`) and plane image chips (see `plane_chips.jsim`).
The plane image chip simulation is included below with a brief discussion:

.The `plane_chips.jsim` simulation configuration.
[source,json]
----
include::plane_chips.jsim[]
----

The output images will all be 180 x 180 with the GSD ranging from 0.25 to
0.35 meters with a PAN channel.  The time range is set to 100 - 120 seconds
so that we capture the plane during the touchdown of the landing sequence.
The integration time is set to 10 ms (0.01 seconds) to capture any motion
blur of the plane.
The output images will be named `chip00.img`, `chip01.img`, etc.
In addition to the single radiance channel, the output image files will
include 4 "truth" channels that include the 
http://www.dirsig.org/docs/new/truth.html#SceneEnuXyz[Scene XYZ]
for each pixel and the 
http://www.dirsig.org/docs/new/truth.html#GeometryIndex[geometric index]
(which can use used to resolve which instance is captured in each chip).  
The camera
can look at the targets from 10 to 30 degrees off-nadir and anywhere in
azimuth. Similarly, the source (sun) can illuminate the scene from 10 -
40 degrees off nadir and from 80 - 280 degrees azimuth.  The output image
set will consist of 25 images and the target set will be anything that is.
The (optional) label report file will be saved as `plane_labels.txt`.
The `hide_others` option has been specified, which means that when imaging
a given target in the selection set, all other targets in that set will
be hidden.  The `make_meta` option has also been specified, which means
that for each image chip there will be a JSON formatted meta-data file
produced that captures the key parameters in the image.

=== Generating plane images
To run the simulation, start the DIRSIG5 program on the command-line
with the `plane_chips.jsim` file:

----
$ dirsig5 plane_chips.jsim
----

As the model runs, a short message will appear regarding the number of
instances matched by the requested tags.  In this case, it correctly found
2 targets (one "parked" and one "flying").  Since we are generating
25 images, we will image several of these targets many times.  As each
image chip is generated, a small output message will be written to the
output with the random parameters used for that chip.  The "target index"
indicates which of the 2 targets was selected for that image.

----
Plugin status:
    Found 56 installed plugin(s)
    Found 0 user plugin(s)

User scene(s):
   /Users/dirsig/demos/ChipMaker2/demo.scene.hdf

Loading Plugins:
Setting up the 'ChipMaker' plugin

Requesting instances with tag(s):
    plane
Checking for scene tags:

Summary of scene tags:

Primary scene location: 43.0000000000 +N, 78.0000000000 +W, 0.0000000000 [m]

Running in multi-threaded mode with 10 threads

Initializing the scheduler

Initializing the 'ChipMaker' plugin:
    Random seed = 1
    Using convergence parameters = 20, 100, 1.000e-06
    Using a maximum of 4 nodes per path
    Checking target tag requests:
        'plane': found 2 instance(s)
        2 total target(s) found
    Image chip count = 25
        Image size = 180 x 180
        Image basename = chip
    Truth requests:
        Scene ENU X Coordinate [m]
        Scene ENU Y Coordinate [m]
        Scene ENU Z Coordinate [m]
        Geometry Index
    Pre-computing chip parameters ...

Starting chip 1 of 25
    Image filename = 'chip18.img'
    Target index = 0
    Tags = 'aircraft', 'plane', 'badger', 'parked'
    Keypoints = left_wing, nose, tail, right_wing
    Location = -90.000, 375.000, 0.000
    Offset = 2.937, 2.669, 0.000
    Time = 102.212 [seconds]
    GSD = 0.289 [meters]
    View angles = 16.26, 237.33 [degrees]
    View distance = 1000.000 [meters]
    Source angles = 27.11, 209.80 [degrees] 
----


== Results ==

=== Plane images
The image below is an example chip produced by the `plane_chips.jsim`
scenario.  

.An example image with meta-data overlays from the "plane" selection set.
image::images/plane.png[width=512]

Below is the JSON meta-data file for the above image, which captures
the values for all the random variables used for this image are
captured here.  The overlaid polygon (purple) captures the extents
of the plane target object (see the `instance_polygons` object in
the JSON).  The overlaid points result from the
Badger bundle including 3D "<keypoints>" (discussed previously),
the 2D `keypoints` object is included with the corresponding 2D
image coordinates for those 3D points in the image.  Note that the
"right wing" has an X coordinate of `-1` indicating that this point
is outside the image.

.The corresponding JSON meta data file for the image.
[source,json]
----
include::images/plane.meta[]
----

The grid below shows a series of chips of the plane produced by a single
simulation.

.All the images for the "planes" selection set.
image::images/plane_chips.png[width=512]

The other key output is the label report, which contains a comma-separated
row of parameters for each image chip. Below is the `plane_labels.txt` file:

----
# DIRSIG5 'ChipMaker' label report
# Generated on Fri Mar 15 11:53:10 2019
#
# Input parameters:
#     Random seed = 650
#     Image size = 180 x 180
#     Time range = [100:120]
#     GSD range = [0.25:0.35]
#     View zenith range = [10:30]
#     View azimuth range = [0:360]
#     View offset range = [0:0]
#     Source zenith range = [10:40]
#     Source azimuth range = [80:280]
#
# Fields:
# targetIndex, X, Y, Z, time, GSD, viewZen, viewAzi, sourceZen, sourceAzi, tags
0,0,-90,375,0,117.599,0.278749,22.4733,232.082,34.8964,198.119,"parked:badger:plane:aircraft:active"
1,1,262.892,-336.043,57.0948,119.118,0.324486,13.3063,304.117,25.2511,131.569,"flying:badger:plane:aircraft:active"
2,0,-90,375,0,118.016,0.263938,28.2886,277.251,31.7074,143.393,"parked:badger:plane:aircraft:active"
3,0,-90,375,0,109.299,0.328899,25.3204,224.652,24.6061,116.35,"parked:badger:plane:aircraft:active"
4,0,-90,375,0,104.774,0.330986,29.432,70.0678,30.8078,81.5123,"parked:badger:plane:aircraft:active"
5,1,262.892,-282.857,49.2271,114.327,0.254662,12.535,179.402,19.119,130.689,"flying:badger:plane:aircraft:active"
6,1,262.892,-240.515,41.6136,110.494,0.309903,24.9769,0.106731,33.9668,146.441,"flying:badger:plane:aircraft:active"
7,0,-90,375,0,117.613,0.278681,13.9235,49.2909,39.3508,145.493,"parked:badger:plane:aircraft:active"
8,0,-90,375,0,107.964,0.328685,19.3926,166.087,32.2435,126.778,"parked:badger:plane:aircraft:active"
9,1,262.892,-228.804,39.294,109.43,0.320495,10.9795,256.394,11.2589,83.732,"flying:badger:plane:aircraft:active"
10,1,262.892,-178.982,28.3761,104.886,0.263345,21.9115,153.293,13.336,265.872,"flying:badger:plane:aircraft:active"
11,1,262.892,-339.21,57.5044,119.402,0.304839,17.2521,339.849,35.7509,184.613,"flying:badger:plane:aircraft:active"
12,0,-90,375,0,103.552,0.311723,27.763,223.276,35.1643,82.0975,"parked:badger:plane:aircraft:active"
13,0,-90,375,0,116.552,0.295408,25.7081,341.612,11.244,129.288,"parked:badger:plane:aircraft:active"
14,1,262.892,-336.045,57.0951,119.118,0.333699,11.714,30.928,27.4111,112.683,"flying:badger:plane:aircraft:active"
15,0,-90,375,0,115.016,0.281598,17.1845,282.722,37.0417,223.038,"parked:badger:plane:aircraft:active"
16,0,-90,375,0,105.576,0.304214,13.111,208.036,23.4894,97.3484,"parked:badger:plane:aircraft:active"
17,0,-90,375,0,113.957,0.301116,22.6629,78.7787,13.8918,128.61,"parked:badger:plane:aircraft:active"
18,1,262.892,-156.986,23.0085,102.868,0.313683,24.1875,36.2972,39.6899,100.067,"flying:badger:plane:aircraft:active"
19,1,262.892,-211.173,35.6254,107.826,0.253148,17.4392,339.537,12.1514,271.57,"flying:badger:plane:aircraft:active"
20,0,-90,375,0,106.708,0.346194,12.0769,316.948,10.1301,157.73,"parked:badger:plane:aircraft:active"
21,1,262.892,-287.496,49.9881,114.746,0.261275,26.4576,289.819,13.5948,184.798,"flying:badger:plane:aircraft:active"
22,1,262.892,-287.394,49.9715,114.737,0.250107,24.4218,55.9667,12.2976,225.202,"flying:badger:plane:aircraft:active"
23,0,-90,375,0,101.357,0.25031,23.78,104.247,35.1732,273.617,"parked:badger:plane:aircraft:active"
24,1,262.892,-257.061,44.7325,111.994,0.311358,13.4397,207.051,21.5705,146.608,"flying:badger:plane:aircraft:active"
----

=== Chopper images
The image below is an example image from the "choppers" selection set
produced by the `chopper_chips.jsim` simulation file with the extent 
polygon from the JSON meta-data file overlaid.

.An image with meta-data overlay from the "choppers" selection set.
image::images/chopper.png[width=512]

The image grid below features 25 image chips produced by the
`chopper_chips.jsim` simulation file.

.The images for the "choppers" selection set.
image::images/chopper_chips.png[width=512]

