ABSTRACT (original article): Many flies and moths mimic the frontal appearance of jumping spiders. This type of mimicry, which we term as partial mimicry, can be distinguished from Batesian mimicry since the mimic has spider resembling patterns only in certain parts of the body, and not the entire body. The presence of spider-like patterns is obvious only at certain angles suggesting that the mimic is frequently targeted by its predators from particular angles. We tested this hypothesis using Deep Convolutional Neural Networks (DCNNs). First, we trained the network on images of forward facing jumping spiders, where features such as the large principal eyes, small lateral eyes and outstretched legs were evident. Then we tested the classifier on images of jumping spider mimicking flies and moths. A probability value according to the likelihood of the image being a jumping spider or not was assigned by the classifier. We show that the classifier was more likely to misidentify mimicking flies and moths as jumping spiders, but that this probability varied according to the species tested. We further tested it on images of flies from different angles and by taking into consideration the visual acuity of potential predators. Our results suggest that neural networks can be efficient tools for testing evolutionary hypotheses, and that partial mimicry may be a result of the effect of the signaling angle and orientation of the mimics in combination with the likelihood that predators may depend on cognitive shortcuts to identify insects as prey. Further experiments incorporating the properties of the visual system of predators (such as vision in ultraviolet) would result in a better understanding of the evolution of partial mimicry. CITATION (original article): Carlos E. Muñoz-Amezcua, Horacio Tapia-McClung, Dinesh Rao (2023), When a glimpse is enough: Partial mimicry of jumping spiders by insects, Behavioural Processes, 213, 104956. https://doi.org/10.1016/j.beproc.2023.104956
Image selection and pre-processing

Overview

All images were obtained from INaturalist research-grade observations. Jumping spider records were obtained through GBIF with a search for INaturalist observations, while non-spider images were obtained both through GBIF and from INaturalist using the search functionality. The selection of images used for this study was first done through a large random sample, where images with certain characteristics (e.g. forward facing jumping spiders) were chosen. After finding that some images had licenses restricting their use, they were removed and in some cases new images were selected.
This process includes many processing steps that were done using directories not in the github repository, so going through all these steps will most likely result in a different set of images. Run the code in NeuralNetTraining.nb and PM-analysis.nb, with the prepared dataset to replicate the results shown in the paper.

Data

Records for INaturalist research grade observations should be stored in the directory given by evaluating the following cell, and should look like “ProjectDirectory/Data” (with “ProjectDirectory” being the directory where this project is cloned):
In[]:=
dataDirectory=FileNameJoin[{DirectoryName@NotebookDirectory[],"Data"}]
Out[]=
/home/carlos/Shared/Projects/Data
If this code and data set was obtained through figshare (https://figshare.com/s/9ba010e0fa843b74402f ), the GBIF occurrence datasets will need to be first extracted into the Data directory. If instead it was obtained through GitHub (https://github.com/CarlosErnestoMA/PartialMimicryWithMachineLearning ), you will have to download the GBIF occurrence datasets using the DOIs given in the ReadMe file (because of limits to the size of files in GitHub). If downloading the GBIF records using the DOIs, save them as “GBIFSalticidaeRecordsData.zip” and ”NonSpiderRecords.zip”.
The INaturalist search results, as well as all training images, are available through both figshare and GitHub in compressed format.

Occurrence records and INaturalist search results

(*filenames*)​​dataDirectory=FileNameJoin[{DirectoryName@NotebookDirectory[],"Data"}];​​zippedRecordsFileNames=FileNameJoin[{dataDirectory,#}]&/@{"GBIFSalticidaeRecordsData.zip","NonSpiderRecords.zip","NonSpiderINaturalistSearchResults.zip"};​​(*Checkthatthezippedfilesareintherightplace*)​​AllTrue[zippedRecordsFileNames,FileExistsQ]
Out[]=
True
Extract them with:
In[]:=
With[​​{nonSpiderRecords=FileNameJoin[{dataDirectory,"NonSpiderRecords.zip"}],spiderRecords=FileNameJoin[{dataDirectory,"GBIFSalticidaeRecordsData.zip"}],inatSearch=FileNameJoin[{dataDirectory,"NonSpiderINaturalistSearchResults.zip"}]},​​ExtractArchive[inatSearch,dataDirectory];​​ExtractArchive[nonSpiderRecords,FileNameJoin[{dataDirectory,"NonSpiderRecords"}],CreateIntermediateDirectories->True];​​ExtractArchive[spiderRecords,FileNameJoin[{dataDirectory,"GBIFSalticidaeRecordsData"}],CreateIntermediateDirectories->True];​​]
Check that the files were extracted correctly
Print["Non spider files extracted: ",DirectoryQ[FileNameJoin[{dataDirectory,"NonSpiderRecords"}]]];​​Print["Spider files extracted: ",DirectoryQ[FileNameJoin[{dataDirectory,"GBIFSalticidaeRecordsData"}]]];​​Print["INaturalist search results extracted: ",FileExistsQ[FileNameJoin[{dataDirectory,"NonSpiderINaturalistSearchResults.csv"}]]];
Non spider files extracted: True
Spider files extracted: True
INaturalist search results extracted: True

Jumping spider image selection

Raw data

Raw data was obtained from GBIF (https://www.gbif.org/), from a search for all records withing the taxon Salticidae and an INaturalist research grade observation (DOI: https://doi.org/10.15468/dl.af78fw ). The resulting Darwin Core Archive (DwC-A), containing details of observations (including URL to the images included in the observation) were processed in this notebook.

Initialization


Selecting records and downloading data

Note that downloading the the intermediate files (raw images and the results of using LabelMe to label and crop them) was done using directories not necessarily included in the structure of the github repository, so if you want to try it out you will have to replace some directories with ones specific to your setup.

Import the DwCA and select a random sample of records

Use importDwCA to import records. It could take around 15-30 seconds.
In[]:=
Print["Time to import: ",First@AbsoluteTiming[allSpiderRecords=importDwCA["GBIFSalticidaeRecordsData"];]," seconds."];​​Length[allSpiderRecords]
Time to import: 23.1815 seconds.
Out[]=
112665
Select images with the appropriate licences:
In[]:=
unusableLicence="http://creativecommons.org/licenses/by-nc-nd/4.0/";​​usableRecords=Select[allSpiderRecords,Not@MemberQ[#[["ImageData",All,"License"]],unusableLicence]&];​​Length[usableRecords]
Out[]=
111095
Obtain a random sample with a wide taxonomic range:
◼
  • Most records are in the genus Phiddipus, while most genus have very few records. To get a wide taxonomic range, the maximum number of records for a genus is capped at 50. Since the 20 most represented genus have more than that (the 20th most common genus has more than 1500 records), this is more than enough to get >1000 images (which will then be reduced after curation).
    • In[]:=
    ReverseSort[Counts@Values[usableRecords[[All,"genus"]]]][[;;20]]
    Out[]=
    Phidippus23222,Salticus8060,Platycryptus7795,Menemerus7641,Anasaitis3528,Habronattus2965,Plexippus2789,Paraphidippus2676,Colonus2371,Lyssomanes2320,Hentzia2217,Marpissa2070,Hypoblemum1982,Evarcha1789,Eris1669,Philaeus1622,Hasarius1617,Maevia1602,Pelegrina1575,Sitticus1522
    ◼
  • Start with a sample set with up to 50 records per genus
    • In[]:=
    spiderGenera=Union[Values[usableRecords[[All,"genus"]]]];​​largeSpiderSampleSet=Join@@Table[​​RandomSample[Select[usableRecords,#["genus"]==genus&],UpTo[50]],​​{genus,spiderGenera}​​];​​Length[largeSpiderSampleSet]
    Out[]=
    7382
    ◼
  • Then select a sample from that set with 1k records (with possibly more images, as an observation can contain multiple images)
    • finalSpiderSample=RandomSample[largeSpiderSampleSet,1000];
    ◼
  • Genera included in sample
    • In[]:=
    ReverseSort@Counts[finalSpiderSample[[All,"genus"]]]//Short
    Out[]//Short=
    Mexigonus15,Pellenes13,Jotus12,196,Bristowia1,Tulpius1,Orienticius1
    ◼
  • Total number of images in the sample
    • In[]:=
    Total[Length/@finalSpiderSample[[All,"ImageData"]]]
    Out[]=
    2327

    Download images

    Specify the directory where the raw, uncropped images will be stored.
    imageDataDir=;
    In[]:=
    importImage[record_]:=Module[{gbifID,imagefileDestination,imageFormat},​​gbifID=record["gbifID"];​​MapIndexed[​​(imageFormat="."<>StringDelete[#1["Format"],"image/"];​​imagefileDestination=FileNameJoin[​​{imageDataDir,gbifID<>"Image"<>ToString[#2[[1]]]<>imageFormat}​​];​​CopyFile[URL[#1["ImageURL"]],imagefileDestination,OverwriteTargetTrue])​​&,​​record["ImageData"]​​]​​]
    testSpiderSample was used to verify that things are working as expected
    In[]:=
    testSpiderSample=RandomSample[finalSpiderSample,10];
    In[]:=
    Length[finalSpiderSample]
    Out[]=
    1000
    In[]:=
    With[​​{recordCount=Length[finalSpiderSample]},​​i=0;​​PrintTemporary["Downloading images for ",recordCount," records..."];​​Monitor[​​Do[​​i+=1;​​importImage[record];​​,​​{record,finalSpiderSample}​​],​​"Importing record "<>ToString[i]​​]​​]

    Image selection and cropping

    Images were downloaded into an intermediate directory, then, using labelme (https://github.com/wkentaro/labelme ), a bounding box was added around the a spider and labeled it accordingly. Images without a clear frontal view of a spider were discarded. The result was a list of files containing the images, the label, and the coordinates of each bounding box. The following code was then used to extract an imaged by cropping the original image using the bounding box.
    ◼
  • List of all json files with the bounding boxes of spider images

    • Non-spider image selection

    All non spider images are from INaturalist research grade observations. A first set of images were selected, then, to increase the training data set for non spiders, more images were selected using an INaturalist search as the GBIF data is very large and slow to process (it takes up about 5.5GB of memory when loaded into Mathematica).
    Download the Darwin core archive, available by going to https://doi.org/10.15468/dl.s5nxft (or provided in this dataset if obtained through figshare). Make sure it has been extracted and placed in the directory shown when evaluating the following cell:

    Select and process images from the GBIF dataset

    Select up to 50 records for a given genus

    ◼
  • Total number of images in the sample

    • Remove images with unusable licences

    Attributions

    Attributions for images are exported into the file “TrainingImageAttributions.tsv”
    The first line is the header, with the following columns names:
    * ImagesID: The ID used to name a set of images from an observation.
    * rightsHolder: The username of the rights holder, as specified in the INaturalist observation.
    * license: The license associated with the images.
    * references: The url to the INaturalist observation containing the images associated to the attribution data.

    Prepare and export attributions

    The attributions for the images used for training the neural network are included in a CSV file with the following columns:
    ImagesID: The ID used in the filenames of the images used for training.
    rightsHolder: The name of the author as specified by the source.
    license: The license which was applied to the image in use.
    references: The url of the observation containing the image.
    Training the spider-or-not classifier
    This chapter contains the code used for training a neural network to classify images of arthropods into the categories “Spider” and “Non-spider”.

    Notes on running this code:

    ◼
  • The initialization cells (grey cells) are automatically run when accepting a prompt asking if you want all initialization cells run after running any cell in the notebook for the first time. The code in those cells prepare the training and validation sets. After those are run you can jump to the section “Use transfer learning to train the network” and follow the instructions there.
    • ◼
  • To ensure reproducibility, all code cells containing a function that uses randomness, the random number generator is re-set using a pre-generated random number as a seed.
    • ◼
  • The system OS and Mathematica version in which this code was run is
    • OperatingSystem: Unix
    System: Linux x86 (64-bit)
    Mathematica version: 13.1.0 for Linux x86 (64-bit) (June 16, 2022)

    Training images

    These are in both GitHub (https://github.com/CarlosErnestoMA/PartialMimicryWithMachineLearning ) and figshare (https://figshare.com/s/9ba010e0fa843b74402f ) in compressed format. You can uncompress them manually, or run the following code:
    Check that the training images have the directories for Spider and NonSpider, and each one contains a list of image file names:

    Use transfer learning to train the network

    The neural network used here was obtained through the Wolfram neural net repository: https://resources.wolframcloud.com/NeuralNetRepository/resources/EfficientNet-Trained-on-ImageNet/
    ◼
  • Take all but the last two layers of EfficientNet
    • ◼
  • Add two layers and a decoder which decodes into two categories, one for Spider and one for NonSpider
    • ◼
  • Train the net. Use the commented code if there is a GPU available and you want to use it, although it won’t take much time without one. (It took around 10 minutes in an average laptop without GPU). Run the un-commented code to replicate the results shown in this notebook.
    • ◼
  • Store the trained network in a .m file:
    • ◼
  • Check accuracy of the network as the proportion of images classified accurately and visualize it in a ConfusionMatrixPlot:
    • ◼
  • Visualize the worst classified examples:

    • Explore the features used by the trained network to identify spiders vs non-spiders:

    Following https://www.wolfram.com/language/12/neural-network-framework/visualize-the-insides-of-a-neural-network.html, the following code can be used to produce images which the DCNN classifies as Spiders or NonSpiders with high confidence:

    Creating a random image and modify it to look like a spider (to the network)

    The first image (a randomly generated image) is classified as a non spider a relatively low confidence:
    While the last image, which has been modified, is now classified as a spider with a really high confidence:
    After 5 cycles the confidence is almost 1.

    Creating a random image and modify it to look like a non-spider (to the network)

    Partial mimicry analysis

    Import Network

    Tests

    0. Does the net work?

    The classifier should accurately identify a jumping spider, with higher accuracy for a front facing vew.

    1. Proof of concept with Rhagoletis pomonella and Ceratitis capitata

    Rhagoletis pomonella and Ceratitis capitata are flies which are believed to mimick jumping spiders -we expect them to be classified as a jumping spider with higher confidence compared to a “tipical” house fly (the rightmost image).

    2. Other salticid mimicking insects

    This tests the classifier using other insects believed to mimick jumping spiders:

    3. An example fly species from different angles

    Partial mimicry is different from other types of mimicry because only part of an organism resembles the target. In this case, a Ceratitis fly resembles a jumping spider from a specific angle where eye spots and wing markings resembling spider legs are visible. We expect this classifier to classify an image as a spider where angles where these features are more prominent:

    4A. Incorporating visual acuity (uncorrected,5, 8, 10 cm distance) Wasp visual system with cpd of 1.2

    In a natural setting, a predator, such as a wasp, will not percieve a mimic with the full resolution of the images we present to the classifier. To better represent a target organism’s visual acuity, we present an a series of images to the classifier with a reduced resolution corresponding to different distances of the mimic to its target:

    4B. Incorporating visual acuity (uncorrected,5, 8, 10 cm distance) jumping spider visual system with cpd of 3.86 (phidippus)

    Jumping spiders have higher visual acuity compared to wasps, these images represent how a jumping spider from the genus Phidippus views a fly from the same set of distances as shown previously:

    Photo Links

    https://www.flickr.com/photos/nickadel/34870124860/​
    ​https://www.flickr.com/photos/nickadel/37205626390/
    image1 https://twitter.com/singaporemacro/status/1358611723133935616?s=20&t=fz0NguYLfrG2Ug6IkVDelA​
    image2 https://commons.wikimedia.org/wiki/File:Olethreutes_arcuella_side.jpg​
    image3 https://twitter.com/elaphrornis/status/1603530324918652932?s=20&t=fz0NguYLfrG2Ug6IkVDelA​
    image4 https://www.flickr.com/photos/pasha_k/16891823956​
    image5 https://inaturalist.ca/observations/14685506​
    image 6 https://www.brisbaneinsects.com/brisbane_planthoppers/AcaciaHopper1.htm

    CITE THIS NOTEBOOK

    When a glimpse is enough: partial mimicry of jumping spiders by insects​
    by Carlos E. Muñoz-Amezcua, Horacio Tapia-McClung, Dinesh Rao
    Wolfram Community, STAFF PICKS, April 17, 2024
    ​https://community.wolfram.com/groups/-/m/t/3159862