# ChipMaker2

Keywords: chips, machine learning

## Summary

This is an advanced demonstration of using the ChipMaker combo plugin. In order to facilitate robust training of machine learning algorithms, we want a large number of images featuring the content of interest. These image "chips" should span a wide range of acquisition parameters. This plugin was created to streamline the generation of image chips with "labels" by automating the sampling of a user-defined parameter space in a single simulation rather than relying on an infrastructure of external scripts to iterate through the parameter space.

Important | This is a DIRSIG5 specific feature and cannot be used in DIRSIG4.

## Details

In order to facilitate robust training, we want to generate a large number of image chips across a wide range of acquisition parameters. Those include:

- Different targets and/or variants of targets

- Different backgrounds (the context within which the target appears)

- Different ground sampling distances (GSDs)

- Different sensor view angles (zenith and azimuth)

- Different illumination angles (zenith and azimuth)

Historically this has been accomplished using external scripting with a conventional DIRSIG simulation. The primary goal of this plugin is to make it easy to configure all the degrees of freedom in one location and have the plugin manage the creation of the images.

## Important Files

This section highlights key files important to the simulation.

### Scene

The scene was developed specifically for this demo. Although the scene is very simple, it illustrates techniques that can be used to develop other scenes appropriate for use in this algorithm training applications. The goal of the scene construction approach at this time is to create a large number of contexts for each target of interest. In this case, the context primarily refers to the background. To create a robust training set for a given target, we want images of that target against a large number of backgrounds. There are currently two ways to accomplish this with the current DIRSIG workflow:

- Create a scene with a divergse background and create a large number of instances of the target within that scene. Each instance of the target will be in a unique location and, hence, have a unique background. In this demo, the helicopters use this approach.

- Create a scene with a diverse background and use motion of a small set of target instances that are in motion. When the targets are imaged at different times, there will be a unique background around the object. In this demo, the bomber plane uses this approach.

#### The Background

The background for the scene is defined as a flat terrain with a material (class) map defining asphalt, concrete, grass and dirt with texture maps. This object is defined in a bundle (see `geometry/bundles/background`).

#### The KA-27 Helicopters

The KA-27 helicopter is defined in a set of bundles. There are 6 total variants of the KA-27: 3 materials variants (Indian, Korean and Russian) and 2 state variants (idle and active). The material variants use a unique set of materials for each vehicle (different reflectance maps featuring nation specific paint schemes and markings). The 2 state variants define if the rotors on the vehicles are in motion or not. This results in 6 permutations of the 3 materials and 2 states. These are captured in the 6 GLIST files within the KA-27 bundle folder:

- `geometry/bundles/ka27/ka27_indian_active.glist`

- `geometry/bundles/ka27/ka27_indian_idle.glist`

- `geometry/bundles/ka27/ka27_korean_active.glist`

- `geometry/bundles/ka27/ka27_korean_idle.glist`

- `geometry/bundles/ka27/ka27_russian_active.glist`

- `geometry/bundles/ka27/ka27_russian_idle.glist`

In order to select these objects in the ChipMaker, the target objects have the tags defined at the top-level geometry list in the bundle GLIST files. The example below is from the `ka27_russian_idle.glist` file showing the tags defined:

    <geometrylist tags="ka27,choppers,russian,idle,aircraft">
      <object search_paths="local">
        <localmaterials>materials/chopper.mat</localmaterials>
        <basegeometry>
          <obj>
            <filename>ka27.obj</filename>
            <assign id="ka27_body_russian">body</assign>
          </obj>
        </basegeometry>
        <staticinstance/>
      </object>
    </geometrylist>

#### The "Badger" Bomber Plane

The "Badger" bomber is defined in a bundle (see `geometry/bundles/badger`). There is only a single instance of the bomber in the scene, which is defined in the `geometry/planes.glist` file.

In order to select these objects in the ChipMaker, the target objects have the tags defined at the top-level geometry list in the bundle GLIST files. The example below is from the `badger.glist` file showing the tags defined:

    <geometrylist tags="badger,plane,aircraft">
      <object enabled="true" search_paths="local">
        <localmaterials>materials/badger.mat</localmaterials>
        <basegeometry>
          <obj>
            <filename>badger_edited.obj</filename>
            <keypointlist>
              <keypoint name="nose" part="">
                <point><x>0.0</x><y>17.5</y><z>3.2</z></point>
              </keypoint>
              <keypoint name="tail" part="">
                <point><x>0.0</x><y>-20.5</y><z>3.6</z></point>
              </keypoint>
              <keypoint name="left_wing" part="">
                <point><x>-20.40</x><y>-8.45</y><z>2.96</z></point>
              </keypoint>
              <keypoint name="right_wing" part="">
                <point><x>+20.40</x><y>-8.45</y><z>2.96</z></point>
              </keypoint>
            </keypointlist>
          </obj>
        </basegeometry>
        <population>
          <materialvariant>
            <variantid>Body</variantid>
            <distribution>
              <matid weight="0.5">Body1</matid>
              <matid weight="0.5">Body2</matid>
            </distribution>
          </materialvariant>
        </population>
        <staticinstance/>
      </object>
    </geometrylist>

Note that this bundled object also has a set of 3D "keypoints" defined, which correspond to the 3D points for the nose, wing tips and tail in the OBJ file. These 3D points will be reported in the JSON "meta data" files with the corresponding 2D coordinates in the image.

The badger plane is instanced into the scene in two places via the `geometry/planes.glist` file. There is static instance for a parked instance of the plane and a second instance that is is dynamic, using the DeltaMotion model to define a take-off of the plane over the runway in the scene. The total takeoff sequence described in the motion file (see `geometry/motion/PlaneTakeOff.mov`) spans from 0 - 150 seconds. This time window will be used to capture the plane at various states in the take off sequence. Note that we have added the tag "flying" to the dynamic instance. This tag will be added to the tags for this instance of the badger.

    <geometrylist>
      <object enabled="true">
        <basegeometry>
          <glist>
            <filename>bundles/badger/badger.glist</filename>
          </glist>
        </basegeometry>
        <staticinstance tags="parked">
          <translation>
            <point><x>-90</x><y>375</y><z>0</z></point>
          </translation>
          <rotation units="degrees">
            <cartesiantriple><x>0.00</x><y>0.00</y><z>-90.00</z></cartesiantriple>
          </rotation>
        </staticinstance>
        <dynamicinstance tags="flying">
          <motion type="delta">
            <filename>motion/PlaneTakeOff.mov</filename>
          </motion>
        </dynamicinstance>
      </object>
    </geometrylist>

Figure 1. A rendering of the complete scene.

The key feature of this scene is that it utilizes the "tagging" system introduced for DIRSIG5 to add shared and unique "labels" to each of the 21 objects in the scene. These tags will be used in the simulation to select which objects will be sampled in the output image set.

#### Geometry Tags

A more detailed explanation of the tagging system and how tags are inherited is described in the ChipMaker1 demo. Below is a list of the tags in the scene and what targets they map to:

`aircraft`

  Maps to all aircraft (helicopters and planes) in the scene.

`choppers`

  Maps to all the helicopters in the scene.

`planes`

  Maps to all the planes in the scene.

`ka27`

  Maps to all the KA-27 helicopters in the scene.

`badger`

  Maps to all the "Badger" planes in the scene.

`indian`

  Maps to all the Indian aircraft in the scene (in this case, the idle and active KA-27 helicopters).

`korean`

  Maps to all the Korean aircraft in the scene (in this case, the idle and active KA-27 helicopters).

`russian`

  Maps to all the Russian aircraft in the scene (in this case, the idle and active KA-27 helicopters).

`idle`

  Maps to all the idle aircraft in the scene (in this case, the idle KA-27 helicopters).

`active`

  Maps to all the active aircraft in the scene (in this case, the active KA-27 helicopters and the Badger).

`flying`

  Maps to all the active aircraft in the scene that are airborne (in this case, the only the Badger is ever airborne).

## Simulations and Results

### Input Simulation Files

The ChipMaker plugin is configured via a DIRSIG JSIM file. Refer to the plugin manual for specific details about the format and options. This demo includes setups to separately generate helicopter image chips (see `chopper_chips.jsim`) and plane image chips (see `plane_chips.jsim`). The plane image chip simulation is included below with a brief discussion:

The `plane_chips.jsim` simulation configuration.

    [
      {
        "scene_list": [
          {
            "inputs": "./demo.scene"
          }
        ],
        "plugin_list": [
          {
            "name": "ChipMaker",
            "inputs": {
              "camera": {
                "channellist": [
                  {
                    "maximum": 0.8,
                    "minimum": 0.4,
                    "name": "Pan"
                  }
                ],
                "gsd_range": {
                  "maximum": 0.35,
                  "minimum": 0.25
                },
                "image_filename": {
                  "basename": "chip",
                  "extension": "img"
                },
                "image_size": {
                  "x": 180,
                  "y": 180
                },
                "psf": {
                  "image": "gaussian.png",
                  "scale": 10
                },
                "readout": {
                  "integration_time": 0.01
                },
                "truth": [
                  "scene_x",
                  "scene_y",
                  "scene_z",
                  "material_index",
                  "geometry_index",
                  "instance_index",
                  "scene_index"
                ]
              },
              "setup": {
                "count": 25,
                "options": [
                  "hide_others",
                  "make_meta"
                ],
                "random_seed": 1,
                "report_filename": "csv_labels.txt",
                "target_tags": [
                  "plane"
                ]
              },
              "source": {
                "azimuth_range": {
                  "maximum": 280,
                  "minimum": 80
                },
                "zenith_range": {
                  "maximum": 40,
                  "minimum": 10
                }
              },
              "time_range": {
                "maximum": 140,
                "minimum": 100
              },
              "view": {
                "azimuth_range": {
                  "maximum": 360,
                  "minimum": 0
                },
                "offset_range": {
                  "maximum": 3,
                  "minimum": 2
                },
                "zenith_range": {
                  "maximum": 30,
                  "minimum": 10
                }
              }
            }
          }
        ]
      }
    ]

The output images will all be 180 x 180 with the GSD ranging from 0.25 to 0.35 meters with a PAN channel. The time range is set to 100 - 120 seconds so that we capture the plane during the touchdown of the landing sequence. The integration time is set to 10 ms (0.01 seconds) to capture any motion blur of the plane. The output images will be named `chip00.img`, `chip01.img`, etc. In addition to the single radiance channel, the output image files will include 4 "truth" channels that include the Scene XYZ for each pixel and the geometric index (which can use used to resolve which instance is captured in each chip). The camera can look at the targets from 10 to 30 degrees off-nadir and anywhere in azimuth. Similarly, the source (sun) can illuminate the scene from 10 - 40 degrees off nadir and from 80 - 280 degrees azimuth. The output image set will consist of 25 images and the target set will be anything that is. The (optional) label report file will be saved as `plane_labels.txt`. The `hide_others` option has been specified, which means that when imaging a given target in the selection set, all other targets in that set will be hidden. The `make_meta` option has also been specified, which means that for each image chip there will be a JSON formatted meta-data file produced that captures the key parameters in the image.

### Generating plane images

To run the simulation, start the DIRSIG5 program on the command-line with the `plane_chips.jsim` file:

    $ dirsig5 plane_chips.jsim

As the model runs, a short message will appear regarding the number of instances matched by the requested tags. In this case, it correctly found 2 targets (one "parked" and one "flying"). Since we are generating 25 images, we will image several of these targets many times. As each image chip is generated, a small output message will be written to the output with the random parameters used for that chip. The "target index" indicates which of the 2 targets was selected for that image.

    Plugin status:
        Found 56 installed plugin(s)
        Found 0 user plugin(s)

    User scene(s):
       /Users/dirsig/demos/ChipMaker2/demo.scene.hdf

    Loading Plugins:
    Setting up the 'ChipMaker' plugin

    Requesting instances with tag(s):
        plane
    Checking for scene tags:

    Summary of scene tags:

    Primary scene location: 43.0000000000 +N, 78.0000000000 +W, 0.0000000000 [m]

    Running in multi-threaded mode with 10 threads

    Initializing the scheduler

    Initializing the 'ChipMaker' plugin:
        Random seed = 1
        Using convergence parameters = 20, 100, 1.000e-06
        Using a maximum of 4 nodes per path
        Checking target tag requests:
            'plane': found 2 instance(s)
            2 total target(s) found
        Image chip count = 25
            Image size = 180 x 180
            Image basename = chip
        Truth requests:
            Scene ENU X Coordinate [m]
            Scene ENU Y Coordinate [m]
            Scene ENU Z Coordinate [m]
            Geometry Index
        Pre-computing chip parameters ...

    Starting chip 1 of 25
        Image filename = 'chip18.img'
        Target index = 0
        Tags = 'aircraft', 'plane', 'badger', 'parked'
        Keypoints = left_wing, nose, tail, right_wing
        Location = -90.000, 375.000, 0.000
        Offset = 2.937, 2.669, 0.000
        Time = 102.212 [seconds]
        GSD = 0.289 [meters]
        View angles = 16.26, 237.33 [degrees]
        View distance = 1000.000 [meters]
        Source angles = 27.11, 209.80 [degrees]

## Results

### Plane images

The image below is an example chip produced by the `plane_chips.jsim` scenario.

Figure 2. An example image with meta-data overlays from the "plane" selection set.

Below is the JSON meta-data file for the above image, which captures the values for all the random variables used for this image are captured here. The overlaid polygon (purple) captures the extents of the plane target object (see the `instance_polygons` object in the JSON). The overlaid points result from the Badger bundle including 3D "<keypoints>" (discussed previously), the 2D `keypoints` object is included with the corresponding 2D image coordinates for those 3D points in the image. Note that the "right wing" has an X coordinate of `-1` indicating that this point is outside the image.

The corresponding JSON meta data file for the image.

    {
      "atm_conditions": "",
      "gsd": 0.2528943717479706,
      "image_filename": "chip00.img",
      "image_size": [180, 180],
      "instance_polygons": {
        "plane58": [
          [0, 111],
          [0, 126],
          [51, 166],
          [75, 170],
          [77, 170],
          [99, 169],
          [101, 168],
          [158, 136],
          [160, 134],
          [161, 131],
          [162, 126],
          [161, 123],
          [160, 121],
          [90, 31],
          [86, 27],
          [83, 27],
          [0, 111]
        ]
      },
      "keypoints": {
        "left_wing": [159, 129],
        "nose": [85, 29],
        "right_wing": [-1, 119],
        "tail": [75, 167]
      },
      "source_azimuth": 2.2743518352508545,
      "source_zenith": 0.41452136635780334,
      "tags": ["aircraft", "plane", "badger", "parked"],
      "target_index": 0,
      "target_location": [-90.0, 375.0, 0.0],
      "time": 113.15745544433594,
      "view_azimuth": 3.077528715133667,
      "view_distance": 1000.0,
      "view_zenith": 0.3841071128845215
    }

The grid below shows a series of chips of the plane produced by a single simulation.

Figure 3. All the images for the "planes" selection set.

The other key output is the label report, which contains a comma-separated row of parameters for each image chip. Below is the `plane_labels.txt` file:

    # DIRSIG5 'ChipMaker' label report
    # Generated on Fri Mar 15 11:53:10 2019
    #
    # Input parameters:
    #     Random seed = 650
    #     Image size = 180 x 180
    #     Time range = [100:120]
    #     GSD range = [0.25:0.35]
    #     View zenith range = [10:30]
    #     View azimuth range = [0:360]
    #     View offset range = [0:0]
    #     Source zenith range = [10:40]
    #     Source azimuth range = [80:280]
    #
    # Fields:
    # targetIndex, X, Y, Z, time, GSD, viewZen, viewAzi, sourceZen, sourceAzi, tags
    0,0,-90,375,0,117.599,0.278749,22.4733,232.082,34.8964,198.119,"parked:badger:plane:aircraft:active"
    1,1,262.892,-336.043,57.0948,119.118,0.324486,13.3063,304.117,25.2511,131.569,"flying:badger:plane:aircraft:active"
    2,0,-90,375,0,118.016,0.263938,28.2886,277.251,31.7074,143.393,"parked:badger:plane:aircraft:active"
    3,0,-90,375,0,109.299,0.328899,25.3204,224.652,24.6061,116.35,"parked:badger:plane:aircraft:active"
    4,0,-90,375,0,104.774,0.330986,29.432,70.0678,30.8078,81.5123,"parked:badger:plane:aircraft:active"
    5,1,262.892,-282.857,49.2271,114.327,0.254662,12.535,179.402,19.119,130.689,"flying:badger:plane:aircraft:active"
    6,1,262.892,-240.515,41.6136,110.494,0.309903,24.9769,0.106731,33.9668,146.441,"flying:badger:plane:aircraft:active"
    7,0,-90,375,0,117.613,0.278681,13.9235,49.2909,39.3508,145.493,"parked:badger:plane:aircraft:active"
    8,0,-90,375,0,107.964,0.328685,19.3926,166.087,32.2435,126.778,"parked:badger:plane:aircraft:active"
    9,1,262.892,-228.804,39.294,109.43,0.320495,10.9795,256.394,11.2589,83.732,"flying:badger:plane:aircraft:active"
    10,1,262.892,-178.982,28.3761,104.886,0.263345,21.9115,153.293,13.336,265.872,"flying:badger:plane:aircraft:active"
    11,1,262.892,-339.21,57.5044,119.402,0.304839,17.2521,339.849,35.7509,184.613,"flying:badger:plane:aircraft:active"
    12,0,-90,375,0,103.552,0.311723,27.763,223.276,35.1643,82.0975,"parked:badger:plane:aircraft:active"
    13,0,-90,375,0,116.552,0.295408,25.7081,341.612,11.244,129.288,"parked:badger:plane:aircraft:active"
    14,1,262.892,-336.045,57.0951,119.118,0.333699,11.714,30.928,27.4111,112.683,"flying:badger:plane:aircraft:active"
    15,0,-90,375,0,115.016,0.281598,17.1845,282.722,37.0417,223.038,"parked:badger:plane:aircraft:active"
    16,0,-90,375,0,105.576,0.304214,13.111,208.036,23.4894,97.3484,"parked:badger:plane:aircraft:active"
    17,0,-90,375,0,113.957,0.301116,22.6629,78.7787,13.8918,128.61,"parked:badger:plane:aircraft:active"
    18,1,262.892,-156.986,23.0085,102.868,0.313683,24.1875,36.2972,39.6899,100.067,"flying:badger:plane:aircraft:active"
    19,1,262.892,-211.173,35.6254,107.826,0.253148,17.4392,339.537,12.1514,271.57,"flying:badger:plane:aircraft:active"
    20,0,-90,375,0,106.708,0.346194,12.0769,316.948,10.1301,157.73,"parked:badger:plane:aircraft:active"
    21,1,262.892,-287.496,49.9881,114.746,0.261275,26.4576,289.819,13.5948,184.798,"flying:badger:plane:aircraft:active"
    22,1,262.892,-287.394,49.9715,114.737,0.250107,24.4218,55.9667,12.2976,225.202,"flying:badger:plane:aircraft:active"
    23,0,-90,375,0,101.357,0.25031,23.78,104.247,35.1732,273.617,"parked:badger:plane:aircraft:active"
    24,1,262.892,-257.061,44.7325,111.994,0.311358,13.4397,207.051,21.5705,146.608,"flying:badger:plane:aircraft:active"

### Chopper images

The image below is an example image from the "choppers" selection set produced by the `chopper_chips.jsim` simulation file with the extent polygon from the JSON meta-data file overlaid.

Figure 4. An image with meta-data overlay from the "choppers" selection set.

The image grid below features 25 image chips produced by the `chopper_chips.jsim` simulation file.

Figure 5. The images for the "choppers" selection set.