Introduction
Introduction
Historically, runways have always been the safest way to launch and land planes as they allow a vehicle to gradually gain or lose speed. Given the ubiquity of airplanes and their value as fast modes of transportation, the immediate identification of such landmarks can lend itself to be highly useful in areas such as emergency situation response, global surveillance, and aerial reconnaissance. Facilitating this, high-resolution satellite images of nearly every corner of the world have become increasingly available. This project seeks to apply image processing to hundreds of airport satellite images in order to train a neural network capable of identifying an airplane runway from a satellite image.
Data Retrieval and Preparation
Data Retrieval and Preparation
Retrieve Airports and Coordinates
Retrieve Airports and Coordinates
To obtain satellite imagery from the Wolfram Data Repository, the coordinates of several public airports are needed as center anchor points for each image within a given zoom level. An entity list of public airports was used to find the list of coordinates. Missing coordinates were filtered out and the final list of coordinates was used to load satellite. The satellite imagery taken from these coordinates will be used as training, validation, and testing sets for the neural network.
Obtain a list of 2724 airports classified as public, which is half of all available airports:
In[]:=
public=EntityList[EntityClass["Airport",{"Uses"->MemberQ["Public"]}]][[1;;2724]];
public[[1;;3]]
Out[]=
,,
Obtain a list of the coordinates of each airport and group each coordinate pair with its corresponding airport:
In[]:=
allCoords=#["Coordinates"]&/@public;coordsAirport=Partition[Riffle[allCoords,public],2];
coordsAirport[[1;;3]]
Out[]=
{39.2898,-94.4424},,{39.4008,-76.9514},,{39.467,-77.0174},
Some coordinates are not available, so these missing values need to be filtered out. 14 of these missing coordinate airport pairs were removed.
Instances of missing coordinates:
coordsAirport[[11;;13]]
Out[]=
Missing[NotAvailable],,Missing[NotAvailable],,{27.2667,105.471},
Only select elements that have numerical and therefore existing coordinates:
In[]:=
existingCoordsAirport=Select[coordsAirport,NumberQ[#[[1,1]]]&&NumberQ[#[[1,2]]]&];
Missing coordinates and their airports have been removed:
existingCoordsAirport[[11;;13]]
Out[]=
{27.2667,105.471},,{66.9511,-53.7292},,{52.185,-128.157},
Retrieve Satellite and Highlighted Runway Images of the Airports
Retrieve Satellite and Highlighted Runway Images of the Airports
Most street map images use a specific grey color which is extracted from the from each airport’s street map to create an interpolation of the runways. This allows for satellite images to be mapped to their corresponding runway images, resulting in datasets that can be used for the training, validation, and testing of a neural network. A universal zoom level of 12.5 was found to fully encompass nearly every runway while still maintaining a level of detail.
A street map image is obtained from the coordinates of each airport at a GeoZoomLevel of 12.5:
GeoImageGeoPosition["Coordinates"],"StreetMapNoLabels",GeoZoomLevel->12.5
Out[]=
A universal grey color is used to represent runways on nearly every street map, so the color distances between each pixel of the street map and the specific grey are found and rounded to create a runway interpretation of a street map.
A runway is extracted from each street map image:
RunwayGrey={0.678431381600601,0.6588229236532062,0.6352932996730977,1.};
,ErosionBinarizeColorDistance
,RGBColor[RunwayGrey],{0,0.01},0.5
Out[]=
,
The processes described above can be combined into a function that receives an airport entity and zoom level as inputs and returns its detected runway based on an analysis on the street map.
The two processes are combined to create a function called RunwayAirport:
In[]:=
RunwayAirport[airport_,zoomLevel_]:=Module[{RunwayWithStreetMap},RunwayWithStreetMap[streetMap_]:=Module[{RunwayGrey={0.678431381600601,0.6588229236532062,0.6352932996730977,1.}},Erosion[Binarize[ColorDistance[streetMap,RGBColor[RunwayGrey]],{0,0.01}],0.5]];RunwayWithStreetMap[GeoImage[GeoPosition[airport["Coordinates"]],"StreetMapNoLabels",GeoZoomLevel->zoomLevel]]]
,RunwayAirport,12.5
Out[]=
,
RunwayAirport[] is applied to find the runway image of each airport :
publicRunways=RunwayAirport[#,12.5]&/@existingCoordsAirport[[All,2]];
publicRunways[[49;;51]]
Out[]=
,
,
After the black and white runway images are found, a list of their corresponding airports' satellite images are retrieved at the same level. This allows for an association to be created, mapping each satellite image to its runway image.
A list of every airport's satellite images is retrieved from their coordinates:
In[]:=
getSatMap[airport_,zoomLevel_]:=GeoImage[GeoPosition[airport["Coordinates"]],GeoZoomLevel->zoomLevel]publicSat=getSatMap[#,12.5]&/@existingCoordsAirport[[All,2]];
publicSat[[49;;51]]
Out[]=
,
,
Map each satellite image to its runway image as a set of rules:
Filter Out Unsuitable Satellite and Runway Images
Filter Out Unsuitable Satellite and Runway Images
In many instances, tiles of certain satellite images do not load and return a grey and black checkerboard pattern instead. This may be due to a lack of information at the specified zoom level. These images need to be identified so that they can be filtered out.
Here is an example of a satellite image with a missing section:
In order to decide whether a given satellite image contains the black and grey checkerboard used to represent missing tiles, each color in the image is compared with the black and grey colors and extracted if they are represented within a reasonable maximum color distance of 0.05. The closest black and grey colors are found from the satellite image, and if the color distance between these and the original black and grey colors is below 0.05, it is confirmed that the image contains missing tiles. Otherwise, no matching colors are found and the satellite image remains in the dataset.
The black and grey colors are extracted from the cropped checkerboard image:
Colors from the satellite image that best match the specific grey and black colors are extracted:
Matching black and grey colors are not found if the closest colors are too far from the maximum color distance:
The processes above can be combined to create a function that identifies whether a given satellite image contains a missing checkerboard pattern. This is achieved by checking whether the returned output of FindMatchingColor[] is indeed a color rather than a Missing[] value.
missingIdentifier determines whether the specific black and grey colors are present in the given image:
Satellite images containing missing tiles are now removed from satToRunway so that every satellite image in the dataset is complete within the dimensions.
missingIdentifier is applied to every satellite image in the dataset, and only images that return false are selected:
In addition, some black and white runway images are blank. However, an inspection of its corresponding satellite image reveals that there is in fact a runway present, meaning that the runway image is incorrectly presenting a blank area. This may be due to the usage of a slightly different hue of grey to represent runways on the area's street map. Since these images are relatively uncommon, they are removed from the dataset without severely compromising the amount of data present.
Two examples of blank runway images and their satellite images:
A completely blank image will have image data values that are all 0 (if completely black). The uniformity of values can be found by deleting the repeated values and checking the length of the list of unique values. Thus, a black image will only have the image data value of 0, while an image containing a runway will contain both black and white pixels (values 0 and 1).
Every image data value of a blank runway image is 0:
While normal runway images contain 0s and 1s:
A function is created that identifies blank runway images by checking to see if the length of unique image data values is equal to 1:
Just as satellite images containing missing tiles are removed, completely blank runway images are removed from the dataset.
blankIdentifier is applied to every runway image in the dataset, and runway images that return False are selected:
Neural Network Training and Adjustments
Neural Network Training and Adjustments
The data acquired from the processes above are split into training, validation, and testing groups, with proportions of 70%, 20%, and 10% respectively. The pre-existing Pix2pix Photo-toStreet-Map-Translation neural network is used as a baseline, and transfer learning is implemented to retrain the neural network for a modified purpose. The network was trained three separate times as different problems were identified and solved between different iterations.
Training the Neural Network for the First Time
Training the Neural Network for the First Time
The Pix2pix Photo-toStreet-Map-Translation neural network is trained to convert satellite images into street maps. The functionality is similar to the identification of runways, as both seek to identify and represent road-like structures.
Reformat the data from an association to a list of rules:
Organize data into training, validation, and testing sets:
Import the pre-existing neural network from the Wolfram Neural Net Repository:
Train the neural net with the training and validation data:
It was found that the network accurately identifies runways in the middle of an inputted image, but can not identify runways situated outside the general center area of a satellite image.
Runways are accurately identified when located in the center of the image:
The neural network does not accurately identify runways situated in locations other than the center:
Apply Modifications to the Images
Apply Modifications to the Images
In order to train the neural network to search varying locations of an image rather than being limited to the center, the training and validation datasets needed to be modified so that runways were located in different areas of the images. This was done by cropping each image in one of four ways, which in turn changed the location of each runway to one of the four corners. Random rotations by multiples of 90 degrees were also applied.
The resulting modifier function randomly slices and rotates a pair of satellite and runway images:
The modifier function is applied to each of the datasets:
The datasets are reformatted into a list of rules:
Final Trained Neural Net
Final Trained Neural Net
After two iterations of training and improvements on the image modifiers, a third neural network was trained with the modified datasets. This was able to identify runways situated in any area of the inputted satellite image.
Training the third neural network:
An example usage of the neural network identifying runways in varying locations:
Conclusion and Ideas for the Future
Conclusion and Ideas for the Future
Ideas for Future
Ideas for Future
The accuracy of the neural network could be improved in order to increase its accuracy when differentiating between runways and structures such as rivers and roads. In addition, the project could be extended so that the directions and lengths of each identified runway could be calculated.
Acknowledgements
Acknowledgements
I would like to thank all the mentors at Wolfram High School Summer Camp '21, but I especially thank Stephen Schroeder for guiding me through this project when the light at the end of the tunnel seemed dim. I also want to thank all the TAs, especially Anika, Quinn, and Emma who provided invaluable guidance on neural networks, computational essays, and the art of making a project an enjoyable experience. Finally, I express gratitude towards the Wolfram Data Repository and Wolfram Neural Net Repository which provided the key centerpieces from which I worked off of.