Features and capabilities

Deep learning raster analysis tools enable you to use more than the standard machine learning classification techniques.

• Use convolutional neural networks for image classification and object detection.
• Use a deep learning model to either classify image pixels, detect or classify objects such as airplanes, trees, vehicles, water bodies, buildings, and oil well pads.
• Integrate external deep learning model frameworks, including PyTorch, Keras, TensorFlow, and CNTK.
• Process a collection of rasters during model inferencing to detect change over time or detect objects in different areas of interest.
• Generate a polygon feature class showing detected objects to be used for additional analysis or workflows.
• The deep learning inference tools can use the CPU for distributed processing, or use the powerful GPU(s) on each server node if available.
• Multiple GPUs per server node can be utilized to run deep learning inference job with the model configurations predefined in ArcGIS, including TensorFlow Object Detection API and DeepLab, Keras MaskRCNN, PyTorch fastai Single Shot Detector.
• If the input raster comes with the appropriate map space to image space transform, training image chips can be exported in image space in Export Training Data For Deep Learning.

Get started with deep learning

The creation and export of training samples are done within ArcGIS by a knowledgeable image analyst because they require strong knowledge of image classification workflows. The deep learning model can be trained in ArcGIS using the Train Deep Learning Model raste r analysis tool or ArcGIS API for Python arcgis.learn. Alternatively, the deep learning model can be trained outside ArcGIS using a third-party deep learning API. Once the model is trained, the dlpk item is added to the Portal for ArcGIS and ready for model inference tasks. Run the raster analysis tools to detect and classify objects or classify pixels from Map Viewer, ArcGIS API for Python, ArcGIS REST API, or ArcGIS Pro.

A separate deep learning setup application is also provided, which will install the proper deep learning framework APIs (TensorFlow, Keras, PyTorch and all other required Python libraries) in the ArcGIS Image Server default arcgispro-py3 environment. It is the prerequisite to run deep learning tasks on ArcGIS Image Server

Refer to the help documentation for help configuring and deploying ArcGIS Enterprise for raster analytics. See the additional steps for configuring your Raster Analysis server to utilize multiple GPUs per server node.

1. Create and export training samples.
   1. Use the Label Objects For Deep Learning tool in ArcGIS Pro to select or create a classification schema.
   2. Create training site samples for the class categories or features of interest. Save the training sample file.
   3. Run the Export Training Data for Deep Learning raster analysis tool to convert the source imagery and training samples into deep learning training data. The source imagery may be an image service. The output of the tool is the data store string where the output image chips, labels and metadata files will be stored. An additional output of the tool is a template .emd file to be populated by the data scientist.
2. Train the deep learning model.
   1. Use the image chips generated in step 1 above to train a model using Train Deep Learning Model raster analysis tool. Alternatively, the users can train deep learning models using the ArcGIS API for Python’s arcgis.learn, or third-party deep learning framework APIs.
   2. If the deep learning model is trained using Train Deep Learning Model raster analysis tool, the output will be a deep learning model package item published on the portal. The deep learning model package (.dlpk) can be directly consumed in the next step.
   3. If a third-party deep learning framework API is used to train the model, users need to update the .emd file with the name of the deep learning framework, the model configuration, other optional properties, and optionally a custom Python raster function for model inference. For information on required and optional parameters in the .emd file, see the Esri model definition file section below.
3. Run the inference raster analysis tools from Map Viewer, ArcGIS API for Python, ArcGIS REST API or ArcGIS Pro.
   1. Use the Detect Objects Using Deep Learning, Classify Objects Using Deep Learning, or the Classify Pixels Using Deep Learning raster analysis tools to process your imagery. If your trained model incorporated custom Python raster functions with additional variables such as padding or confidence threshold for fine-tuning the sensitivity, these variables will appear on the ArcGIS Pro or Map Viewer raster analysis tool's dialog box for user input. The data type, such as string or numeric, is specified in the Python raster function.

      The output of the Detect Objects Using Deep Learning tool, or the Classify Objects Using Deep Learning tool, is a feature class showing the objects detected or classified by the model, and the Classify Pixels Using Deep Learning tool outputs a classified raster.

      Note:

      The deep learning model framework must be installed on the Raster Analysis service machines in order to run the inference tools because the inference logic is built into the framework Python API and is required for computation.

Esri model definition file

The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist that trained the model. There are required and optional parameters in the file as described in the table below.

Below is an example of a model definition file (.emd) that uses a standard model configuration:

{
    "Framework": "TensorFlow",
    "ModelConfiguration": "ObjectDetectionAPI",
    "ModelFile":"tree_detection.pb",
    "ModelType":"ObjectionDetection",
    "ImageHeight":850,
    "ImageWidth":850,
    "ExtractBands":[0,1,2],
    "Classes" : [
      {
        "Value": 0,
        "Name": "Tree",
        "Color": [0, 255, 0]
      }
    ]
}

Deep learning package (dlpk) item

Deep learning raster analysis tools require a deep learning model package (dlpk) as input. A deep learning model package is composed of the Esri model definition JSON file (.emd), the deep learning binary model file, and optionally, the Python raster function to be used.

When you have all components ready, you can compress all files into a .zip file, and upload the .zip file to your portal as a dlpk item. For more details, please refer to Add files from your computer. You can also change your .zip file extension to .dlpk so that when adding item, the dlpk item type will be automatically detected.