{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Train a model to identify street signs"},{"metadata":{},"cell_type":"markdown","source":"For support in using these steps in your image classification project, please go to the [Survey123 early adopter community forum](https://earlyadopter.esri.com/project/home.html?cap=e69ef91f45744b98882c651f7b518eb7)."},{"metadata":{},"cell_type":"markdown","source":"This notebook demonstrates how to build and verify a model that can be used to automatically identify street signs using Survey123. It is designed to be used as part of the [Train a model to identify street signs](https://learn.arcgis.com/en/projects/train-a-model-to-identify-street-signs/) Learn ArcGIS tutorial. In the first part of the tutorial, you'll build a survey to collect images that can be used by this notebook to train a model. In the final part of the tutorial, you'll build a survey that uses the model created by this notebook.\n\nThe steps in this notebook include:\n\n* [Set up the environment](#setup)\n* [Download training images from the feature layer](#download)\n* [Train the model](#train)\n* [Test the model (optional)](#test)\n"},{"metadata":{},"cell_type":"markdown","source":"This notebook must be run as an **Advanced with GPU support** notebook. It will consume **30 credits/hour** in ArcGIS Online credits.\nIf you followed the instructions in the [Train a Model](https://learn.arcgis.com/en/projects/train-a-model-to-identify-street-signs/#train-a-model) section of the tutorial, you should be ready to start. You can run each code block in this notebook to train the model.\n\nAnother option is to create a new notebook of your own and copy and paste the code from this notebook into it.  **If** you choose to **create your own notebook and to copy/paste the code**, follow the five steps below, **otherwise, skip to the next cell.** \n\n1. From your ArcGIS Online account click **Notebook**.\n2. Click **New Notebook** and click **Advanced with GPU support**.\n3. Name the new notebook *SignImageCollectionModel_(YourInitals)*.\n4. Give the notebook at least one tag and a description.  \n5. Keep this notebook open in a separate tab or window so you can follow along with the instructions. Remember to save your notebook often."},{"metadata":{},"cell_type":"markdown","source":"## Set up the environment <a class=\"anchor\" id=\"setup\"></a>"},{"metadata":{},"cell_type":"markdown","source":"The first step is to import packages required for the code.  \n1. Click in the cell below and click the **Run** button.  \n\nImporting the packages will take some time."},{"metadata":{"trusted":false},"cell_type":"code","source":"%env ARCGIS_ENABLE_TF_BACKEND=1\nimport os\nimport shutil\nimport sys\nimport tensorflow as tf\n\nfrom pathlib import Path\nfrom arcgis.gis import GIS\nfrom arcgis.learn import prepare_data, FeatureClassifier\n\nfrom PIL import Image\nfrom PIL import ImageFont\nfrom PIL import ImageDraw\n\ntf.keras.backend.clear_session() \ngis = GIS(\"home\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next step is to set a variable to the feature service ID. This tells the model where to get its training data. \n\n> Use the default value in the cell below if you are using the supplied sample geodatabase, or change the feature_service_id variable to match your own feature service ID, if you collected your own training data.\n\n> To identify the ```feature service id``` of your own layer go to ```Add``` on the navigation bar, find the **feature layer** and click the ```Add to Notebook``` button. A cell with sample code with the ID will be added. Copy just the ID and replace the value in the variable-setting statement below. Remove the sample feature service id.\n\n2. Click in the next cell and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"feature_service_id = 'fa40cf680eb4436daf4109b887b52b30'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next step is to set some parameters for the model.\n\n> Use the defaults listed if using the supplied data files, or change to suit your data.  \n\n3. Click in the next cell and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"tmp_dir = '/arcgis/home/tmp'\nnotebook_title = \"SignageClassificationModel\"\nfeature_layer_index = 0\nclassification_field = 'signType'\nmodel_name = 'signs'\nmodel_title = 'Signage Classification Model'\nmodel_description = 'Classify speed and stop signs'\nmodel_tags = 'SignDetection'\nmodel_folder = 'SignDetection'","execution_count":null,"outputs":[]},{"metadata":{"slideshow":{"slide_type":"slide"}},"cell_type":"markdown","source":"## Download training images from the feature layer <a class=\"anchor\" id=\"download\"></a>"},{"metadata":{},"cell_type":"markdown","source":"The first step in this section it to access the feature layer.  \n\n1. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"feature_service = gis.content.get(feature_service_id)\nfeature_service","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, you will print the layer's name and whether it has attached images.  \n\n2. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"feature_layer = feature_service.layers[feature_layer_index];\nprint('layer name:', feature_layer.properties.name)\nprint('has attachments:', feature_layer.properties.hasAttachments)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, you will get all the features from the layer and review the first few in the table head.  \n\n3. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"features = feature_layer.query(return_geometry=False, fields=classification_field)\nfeatures.sdf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, you will go through the features and find the number of categories there are and how many images there are for each category.  \n\n4. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"# get the list of unique feature classes\nfeature_classes = {v:[k] for v, k in enumerate(features.sdf[classification_field].unique().tolist())}\n\n# count the number of images in each class\ntotal_images = 0\nfor i in feature_classes:\n    image_count = len(features.sdf[features.sdf[classification_field] == feature_classes[i][0]])\n    feature_classes[i].append(image_count)\n    total_images += image_count\n\nprint('images per class', feature_classes)\nprint('total:', total_images)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, you will create a directory in your running Notebook environment where the code will download and save the data and images from this feature layer.  \n\n5. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"data_path = Path(os.path.join(tmp_dir, feature_service.id))\nimages_path = Path(os.path.join(data_path, 'images'))\n\nif (os.path.exists(images_path)):\n    print('deleting existing directory:', images_path)\n    shutil.rmtree(images_path)\n\nprint('creating new images directory:', images_path)\nos.makedirs(images_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next you will download the images from the feature layer, then check to ensure the number of downloaded images is correct.  \n\n6. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"# Set initial variables to 0 for counts of images and for image size\nimages_count = 0\nimages_size = 0\n\ndef is_image_attachment(attachment):\n    \"\"\"\n    Determine if the attachment is an image\n    :param attachment: feature attachment\n    \"\"\"\n    contentType = attachment['contentType']\n    return contentType == 'image/png' or contentType == 'image/jpeg'\n\n\ndef resize_image(image_path):\n    \"\"\"\n    Resize the image to 160x160. Crop to the center of the \n    image where necessary. We assume here that the source image\n    will always be larger than the output size.\n    :param str image_path: image path\n    \"\"\"\n    with Image.open(image_path) as img:\n        size = 160\n        width, height = img.size\n        # resize the image to the target dimensions\n        if height > width:\n            img = img.resize((size, round(size / width * height)), Image.BICUBIC)\n        else:\n            img = img.resize((round(size / height * width), size), Image.BICUBIC)\n        # crop the image to the center\n        width, height = img.size\n        x = (width / 2) - (size / 2)\n        y = (height / 2) - (size / 2)\n        cropped = img.crop((x, y, x + size, y + size))\n        cropped.save(image_path)\n    \n\nfor index, feature in features.sdf.iterrows():\n    objectId = int(feature.objectid)\n    attachments = feature_layer.attachments.get_list(oid=objectId)\n\n    # ensure the output folder exists for the image class\n    classification = feature[classification_field]\n    attachments_path = Path(os.path.join(images_path, classification))\n    if not os.path.exists(attachments_path):\n        os.makedirs(attachments_path)\n    \n    # download image attachments to the images folder\n    for a in attachments:\n        if is_image_attachment(a):\n            if not os.path.exists(os.path.join(attachments_path, a['name'])):\n                print(f'downloading ({a[\"id\"]}) {a[\"name\"]}')\n                feature_layer.attachments.download(oid=objectId, attachment_id=a['id'], save_path=attachments_path)\n                resize_image(os.path.join(attachments_path, a['name']))\n            images_count += 1\n            images_size += a['size']\n\nprint('\\ndownloaded:', images_count, \"images\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"7. Click **Files** and browse to the images folder to see downloaded images."},{"metadata":{},"cell_type":"markdown","source":"## Train the model <a class=\"anchor\" id=\"train\"></a>\n"},{"metadata":{},"cell_type":"markdown","source":"Now that you have the training data available, you can start building the model.  \n\nThe following steps use the [arcgis.learn module](https://developers.arcgis.com/python/api-reference/arcgis.learn.toc.html#prepare-data). \n\n> If you get an error message, the problem may be the notebook runtime. Verify that you are using an Advanced Runtime Notebook with GPU support."},{"metadata":{},"cell_type":"markdown","source":"The first step is to divide the training data into **training** and **validation** sets. The default split percentage is 0.1, which means 10% of the data is kept as **validation**.  \n\n1. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"batch_size=16\ndata = prepare_data(\n    path=data_path,\n    dataset_type='Imagenet',\n    batch_size=batch_size,\n    seed=42,\n    resize_to=224\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, display samples of the prepared images in a defined number of rows.  \n\n2. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"data.show_batch(rows=4)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, create a deep learning model for image classification with defined **backbone** and **backend**.  \n\n3. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"backbone = 'MobileNetV2'\nbackend = 'tensorflow'\ndef _supported_backbones():\n    return ['MobileNetV2']\nFeatureClassifier._supported_backbones=_supported_backbones\nmodel = FeatureClassifier(data, backbone=backbone, backend=backend)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, find the **learning rate** of the model.  \n\n4. Click in the cell below and click **Run**.\n\n> This step takes approximately 3 minutes to complete."},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"learning_rate = model.lr_find()\nprint(learning_rate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train the model for the specified number of **epochs**, using the specified **learning rate**.  \n\n5. Click in the cell below and click **Run**.\n\n> This step takes approximately 15 minutes to complete."},{"metadata":{"trusted":false},"cell_type":"code","source":"epochs = 20\nmodel.fit(epochs=epochs, lr=learning_rate)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Plot **validation losses** and **training losses** after training the model.  \n\n6. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"model.plot_losses()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Display the **ground truth** images and results of the corresponding **prediction**.  \n\n7. Click in the cell below and click **Run**."},{"metadata":{"scrolled":true,"trusted":false},"cell_type":"code","source":"model.show_results(rows=2, thresh=0.5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The next step is to save the model with a unique name and publish it to your ArcGIS account.  \n\n8. Click in the cell below and click **Run**.\n\n>If you do not save the model it will be lost when the current session is disconnected.  \n\nThe variable model_name was set equal to 'signs' earlier in this notebook.  The current timestamp will be added to signs to make a unique name."},{"metadata":{"trusted":false},"cell_type":"code","source":"from datetime import datetime\ntimestamp = datetime.now().strftime('_%d%m%Y_%H%M%S')\nmodel_name2 = model_name + timestamp\nmodel_path = model.save(model_name2, framework=\"tflite\", publish=True)\nmodel_path","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, update the model metadata.  \n\n9. Click in the cell below and click **Run**.\n\nIf you get an error, please try increasing the number of ```max_items``` in ```gis.content.search```, where it says ```max_items=1000```."},{"metadata":{"trusted":false},"cell_type":"code","source":"model_data_name = model_name2 + '.dlpk'\nmodel_data_path = os.path.join(model_path, model_data_name)\nmodel_item_properties = {\n    'type': 'Deep Learning Package',\n    'title': model_title + timestamp,\n    'description': model_description + \". \" + \"File Database: \" + feature_service.title,\n    'tags': model_tags\n}\n\n# determine if there is an existing deep learning package with the same name\ndl_packages = gis.content.search('owner:' + gis.properties.user.username, item_type = 'Deep Learning Package', max_items=1000)\n\nmodel_item = None\nfor i in range(len(dl_packages)):\n    if dl_packages[i].name == model_data_name:\n        model_item = dl_packages[i]\n        break\n\n# update the model metadata\nif model_item:\n    print('Updating:', model_item)\n    model_item.update(item_properties=model_item_properties, data=model_data_path)\nelse:\n    print('Adding:', model_data_path, \"folder:\", model_folder)\n    model_item = gis.content.add(item_properties=model_item_properties, data=model_data_path, folder=model_folder)\n    \nmodel_item","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next you will save the configuration of your model building process. This includes:\n\n<ul>\n<li>Model name and dataset name</li>\n<li>Notebook name</li>\n<li>Feature service ID, number of objects, fiel geodatabase, last modified time</li>\n<li>Data information (such as class name and saved path) and batch size</li>\n<li>Model information (backbone and backend)</li>\n<li>Learning rate</li>\n<li>Other notes (you can change or add more)</li>\n</ul>  \n\n10. Click in the cell below and click **Run**.\n"},{"metadata":{"trusted":false},"cell_type":"code","source":"import textwrap\n\nwrapper = textwrap.TextWrapper(initial_indent='\\t', subsequent_indent='\\t')\nstr_data = wrapper.fill(str(data))\n\noutfilepath = str(model_path) + \"/SMTCMR_CONFIG.txt\"\ntext_file = open(outfilepath, \"w\")\ntext_file.write(\n    \"TEST#\" + str(model_name2) + \" - \" + str(feature_service.title)                                 +   \"\\n\" +\n    \"ARGIS NOTEBOOKS: \" + str(notebook_title)                                                    +   \"\\n\" + \n    \"\\n\" + \n    \"ID: \" + str(feature_service_id)                                                               +   \"\\n\" + \n    \"OBJECTS: \" + str(total_images)                                                              +   \"\\n\" +\n    \"FGDB_LAST_MODIFIED: \" + str(datetime.fromtimestamp(feature_service.modified/1000))           +   \"\\n\" +\n    \"\\n\" + \n    \"\\n\" + \n    \"DATA: \"                                                                                     +   \"\\n\" +\n    \"\\t\" + str(feature_classes)                                                                        +   \"\\n\" +\n    str(str_data)                                                                                +   \"\\n\" +\n    \"\\t\" + \"batch_size: \" + str(batch_size)                                                      +   \"\\n\" +\n    \"\\n\" + \n    \"\\n\" + \n    \"MODEL: \"                                                                                    +   \"\\n\" +\n    \"\\t\" + str(backbone) + \", \" + str(backend)                                                   +   \"\\n\" +\n    \"\\n\" + \n    \"\\n\" + \n    \"BEST LR: \" + str(learning_rate)                                                                        +   \"\\n\" +\n    \"\\n\" + \n    \"\\n\" + \n    \"**NOTE\"                                                                                     +   \"\\n\" +\n    \"\\t- The images have been cropped from top & bottom (160 pixels each) into square\"           +   \"\\n\" +\n    \"\\t- All images are checked\"                                                                 +   \"\\n\" +\n    \"\\t- Model is saved as zip file\"\n    )\ntext_file.close()\nprint(\"Config file saved to: \" + str(outfilepath))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, update the ModelFormat parameter to be \"NHWC\". \nThis is a requirement for recent versions of Survey123.\n\n11. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"import json\n\ndef update_emd(new_data, file_name):\n    with open(file_name, 'r+') as file:\n        file_data = json.load(file)\n        if list(new_data)[0] not in file_data:\n            file_data.update(new_data)\n            file.seek(0)\n            json.dump(file_data, file, indent = 4)\n\nmodel_format = {\"ModelFormat\":\"NHWC\"}\nfilename = os.path.join(model_path, ''.join([os.path.basename(model_path),'.emd']))\nupdate_emd(model_format, filename)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next, create a .zip file containing the TensorFlow Lite model and results.  You will download the model and use it in Survey123.\n\n12. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"import shutil\nzip_name = model_path\ndirectory_name = model_path\nshutil.make_archive(zip_name, 'zip', directory_name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Optionally, delete the folder of model and results to save storage space.  \n\n13. Click in the cell below and click **Run**."},{"metadata":{"trusted":false},"cell_type":"code","source":"# Uncomment the two lines below by removing the # signs before each line before running the cell if want to delete the folder\n# import shutil\n# shutil.rmtree(model_path) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"You have completed the training and packaging of the model. All files are stored in the .zip file.  \n\n14. Click **Files**, browse to the model path location, and download the .zip file to save it to your computer.\n\n"},{"metadata":{},"cell_type":"markdown","source":"At this point you can return to the [Test the model](https://learn.arcgis.com/en/projects/train-a-model-to-identify-street-signs/#test-the-model) section of the Learn tutorial, to see how to use these files with Survey123 to test the model in the field.  \n\nIf you choose go back to the Learn tutorial, you can save and close this notebook now. This will stop the notebook from continuing to consume ArcGIS Online credits.  \n\n15. Click Save.  \n\n16.  Close this browser tab."},{"metadata":{},"cell_type":"markdown","source":"## Programatically test the model (optional) <a class=\"anchor\" id=\"test\"></a>"},{"metadata":{},"cell_type":"markdown","source":"The following steps demonstrate how to validate the **accuracy** and **confidence rate** of your model. The last part of the accompanying learn tutorial demonstrates how to test your model in the field with Survey123. When using this model with your own images, you might choose to programatically test the  model first, to gain confidence, before going out into the field. \n"},{"metadata":{},"cell_type":"markdown","source":"1. Create a new folder to hold test data files that we'll download from an image collection."},{"metadata":{"trusted":false},"cell_type":"code","source":"test_path = tmp_dir + \"/test\"\n\nif (os.path.exists(test_path)):\n    print('deleting existing directory:', test_path)\n    shutil.rmtree(test_path)\n\nprint('creating new test data directory:', test_path)\nos.makedirs(test_path)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"2. Set test image collection.\n\n>Use the default listed if using the supplied image collection, or change to match your own image collection. "},{"metadata":{"trusted":false},"cell_type":"code","source":"test_images_collection_id = '67137e0896fa455dabcc78db7e20aa9c'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"3. Retrive metadata about the test image collection."},{"metadata":{"trusted":false},"cell_type":"code","source":"imagecol = gis.content.get(test_images_collection_id)\nimagecol","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"4. Download the test image collection and then extract it into a local folder. "},{"metadata":{"trusted":false},"cell_type":"code","source":"# extract the contents of the zip file into our test images folder\nimport zipfile\nwith zipfile.ZipFile(imagecol.download(), 'r') as archive:\n    for filename in archive.namelist():\n        # ignore MacOS specific metadata folder\n        if not filename.startswith('__MACOSX'):\n            archive.extract(filename, test_path)\n\n# the new folder containing our test images\ntest_images_path = os.path.join(test_path, imagecol.title)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"5. Count the total number of test images in the collection. "},{"metadata":{"trusted":false},"cell_type":"code","source":"# count the number of images in the test folder\nimport glob\nprediction_images_count = 0\nfor file in glob.glob(test_images_path + \"/**/*.jpg\", recursive=True):\n    prediction_images_count += 1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"6. Prepare the dataset of prediction images. Leave the parameters unchanged or you can change/add anything you are familiar with."},{"metadata":{"trusted":false},"cell_type":"code","source":"test_datagen = tf.keras.preprocessing.image.ImageDataGenerator(rescale=1./255)\ntest_images = test_datagen.flow_from_directory(\n    test_images_path, \n    batch_size=prediction_images_count,\n    shuffle=False,\n    target_size=(224,224),\n    interpolation=\"nearest\",\n    color_mode=\"rgb\",\n)\ntest_image, test_label = next(iter(test_images))\nclass_names = [f for f in os.listdir(images_path) if f not in \"models\"]\nprint(class_names)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"7. Run the prediction and save the prediction of each image in your model path."},{"metadata":{"trusted":false},"cell_type":"code","source":"import pandas as pd\ninterpreter = tf.lite.Interpreter(model_path=str(model_path) + \"/\" + str(model_name2) + \".tflite\")\n\ninput_details = interpreter.get_input_details()\n\ninterpreter.resize_tensor_input(input_details[0]['index'], (prediction_images_count, 224, 224, 3))\ninterpreter.allocate_tensors()\n\ninput_details = interpreter.get_input_details()\noutput_details = interpreter.get_output_details()\n\ninterpreter.set_tensor(input_details[0]['index'], test_image)\ninterpreter.invoke()\n\npredictions = interpreter.get_tensor(output_details[0]['index'])\nprint(\"Prediction results shape:\", predictions.shape)\npredFrame = pd.DataFrame(predictions)\npredFrame.columns = class_names\npredFrame.to_csv(str(model_path)+\"/prediction.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"8. Display the result numerically with **accuracy** and **average confidence rate**."},{"metadata":{"trusted":false},"cell_type":"code","source":"import numpy as np\ntCount = 0\nconfLe = 0\ntrueLabel = np.argmax(test_label, axis=-1)\nids = np.argmax(predictions, axis=-1)\nfor n in range(prediction_images_count):\n    if ids[n] == trueLabel[n]:\n        tCount += 1\n        confLe += predictions[n][ids[n]]\nrs = 'Accuracy: ' + str(tCount/prediction_images_count) + '; ' + 'Average Confidence: ' + str(confLe/tCount)\nprint(rs)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"9. Display the result visually showing prediction for each images. Save the figure in your model path."},{"metadata":{"trusted":false},"cell_type":"code","source":"import math\nimport matplotlib.pyplot as plt \n\ntrueLabel = np.argmax(test_label, axis=-1)\nids = np.argmax(predictions, axis=-1)\npredLabels = [numpy.array(class_names)[j] for j in ids]\n\n# create a plot showing each image and the associated prediction\nplt.figure(figsize=(20,9))\nplt.subplots_adjust(hspace=0.5)\ncols = 10\nrows = math.ceil(prediction_images_count / cols)\nfor i in range(len(test_image)):\n    row = math.floor(i / cols)\n    col = i - (row * cols)\n    plt.subplot(rows, cols, i + 1)\n    plt.imshow((test_image[i]).astype(np.float32))\n    color = \"green\" if ids[i] == trueLabel[i] else \"red\"\n    plt.title(predLabels[i], color=color)\n    plt.axis('off')\n_ = plt.suptitle(\"Model predictions (correct: green, incorrect: red)\\n\" + rs)\n\n# save the plot to a file\nplt.savefig(str(model_path)+\"/prediction.png\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"10. Save prediction results in your .zip file. "},{"metadata":{"trusted":false},"cell_type":"code","source":"zip_name = model_path\ndirectory_name = model_path\nshutil.make_archive(zip_name, 'zip', directory_name)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Click **Files**, browse to the nominated location, and download the .zip to save locally."}],"metadata":{"esriNotebookRuntime":{"notebookRuntimeName":"ArcGIS Notebook Python 3 Advanced with GPU support","notebookRuntimeVersion":"4.0"},"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.6.10","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":2}