Tracking ships and visualize them in QGIS
Today I’m going to write about collecting the coordinates, heading, speed and date/time of a ship of interest. After collecting the data I’m going to visualize the data in QGIS.
What am I going to do?
- I’m going to build a tracking device for vesselfinder.com.
- Writing a tool in python
- If you have a Raspberry PI, it’s a perfect device for this project
- Visualize the data in QGIS
Vesselfinder
At the website vesselfinder.com, you can search and track a ship of interest. For example, a suspicious ship that is involved in illegal fishing or smuggling operations. However, it is not possible (free version) to save and export the track of a ship. Let’s do it by ourselves.
The ship I’m going to use for this article is in no way suspicious!!!
For this article, I’m using the ship ‘FREYJA W’. Go to vesselfinder.com and search for the ship name ‘FREYJA W’, click on the name (1) and next the ship will be displayed on a separate page. When clicking on the map you will see the detailed page of the ship.
When (2) clicking on track you can see the track up to a day back in history. We want more! At point 3 we see some additional information like speed and heading. By clicking (3) details we flip to the main page of the FREYJA W.
There is a subsection called ‘POSITION & VOYAGE DATA’. In this subsection, most of the data I’m interested in are presented: (1) course & speed, Coordinates and (3) the timestamp of the received data.
Data retrieval
I’m going to get the data with webscraping. Be careful with this and do NOT abuse it.
I’m going to build a small scraper that will extract the course, speed, coordinates and the update time. The scraper will be built in python3. But before we start coding we need to know where to extract the data, in order to do this I need to view the source of the webpage. So open the source code of the page and search for the subsection ‘POSITION & VOYAGE DATA’.
First,I need to install some extra python packages:
pip install requests
pip install beautifulsoup4
pip install selenium
pip install schedule
After installing the packages we start coding the scraper. Open a new .py file and start importing the following packages:
The scraping process takes place in a function. So let’s create a function:
Line 14 is the creation of the headers for simulating a browser. If we would use python without headers vesselfinder.com will block our request.
Line 17 is the actual URL of the ships page (FREYJA W). At line 18 were going to execute a get request to the ships URL with the given headers. Line 19 is for parsing the received HTML into a variable.
Next, I want to save the data to my local disk. In this way were keeping the traffic as low as possible to the website. Lines 22 & 23 are used for opening and saving the data to HTML file in the same folder as the python program is.
The second part is all about finding the information in the stored document. Lines 28 & 29 are for opening and reading the contents of the HTML file we stored earlier. Do you remember the source code?
The subsection ‘POSITION & VOYAGE DATA’ is stored in a table. And in HTML ‘td’ is used for creating tables. At line 32 we are going to search for all table items and put them into a list. Line 35 extracts the coordinate pair from the list (it is stored in the 22nd item but python lists are zero-based). Line 38 extracts the whole line of code. Line 39 converts the Beautifulsoup object into a string value. Line 40 converts it back to a Beautifulsoup object (not sure why I did this… looks redundant). Line 41 extracts the tooltip information.
Line 44 thru line 46 extracts the course and the speed as a pair. So at line 45, I extract the course aka heading from the pair and at line 46 I extract the speed in knots from the pair.
The final necessary part is all about sanitizing the coordinates and date/time and writing appending it to a CSV file.
The extracted coordinate pair is converted into a string at line 51, after which at line 52 the northing and at line 53 the easting is extracted from the pair.
Then we want to extract and format the date and time stamps. The date and time stamp are as follows:
Mar 23, 2020 12:01 UTC. In order to use this date and time in QGIS, I want to convert it to 2020–03–23 and in a separate field the time 12:01.
At line 56 I remove the comma and the word (including space) UTC. So the current format is Mar 23 2020 12:01. At line 57 I’m using the datetime module to explain the format to the variable dtg. At line 58 and line 59, the date and time are extracted from the dtg variable.
At line 62 I’m calling a global variable. This global variable holds the number of entries. Which are required in QGIS, for creating a track of the ship.
By now it is time to write append all the variables to a CSV file. Line 65 opens a CSV file called AIS_Track.csv in appending mode. Line 66 and 67 are used to write the variables ‘ctr, north, east, date, current_time, heading, speed’ to the CSV file. After this write append action, the file will be closed.
The next part is optional and is not required in order to get the scraper to work. I wanted a screenshot of the website every time the tool retrieved the page for cross reference purposes. For this part I’m going to need a google webdriver, go to https://chromedriver.chromium.org/ and download your webdriver into the same folder as your python file.
At lines 72 and 73 I prepare a prefix filename. The screenshot is taken from the webpage linked to the image (1). Line 76 and 77 is used for extracting the image URL.
Line 80 thru 85 is used to open the google webdriver and get the webpage of the ship. Then it will wait for 5 seconds before it saves a screenshot of the page. After saving the screenshot it will close the webdriver. Line 88 is printing a status message to the python console. And at line 91 the counter will add + 1.
Since the program was written in a function, I need to call the function.
When I start the program the first time it will create the counter, at line 96, which is basically the id number per entry. Next, I will run the function 1 time at line 99. After that, it will re-run the function every 900 seconds (lines 103–106).
Using QGIS to visualize the track of the ship
When the code is put together I let it run for several days. In my case, I recorded the voyage of FREYJA W from England to the Netherlands.
Open the CSV file and add the column headers (ID, North, East, Date, Time (UTC), Heading, Speed). Then close the file. At line 1 you see the column titles, I had to fill in those manually.
Let’s start QGIS and when QGIS is booted, use the (1) ‘QuickMapServices’ plugin to load the (2) Openstreetmap (OSM) data as a background layer.
After loading the OSM layer load the CSV by clicking Layer → Add Delimited Text Layer
The datasource manager will pop-up. Here you can fill in the data in order to load the CSV into QGIS.
(1) Filepath to the CSV file
(2) The Easting column (if everything went well, QGIS will recognize the column)
(3) The Northing column (if everything went well, QGIS will recognize the column)
(4) Click add
(5) And close the data manager
By now we will see a dotted line from north England to Rotterdam, the Netherlands. Although it is imported by now, I want more. I want to see a line and I want to see the heading per point.
Click Processing → Toolbox
The toolbox will appear on the right side of the screen.
To convert the points to a line I’m using the tool ‘Points to path’. So at the processing toolbox (1) fill in at the search bar (2) ‘Points to path’. And double click on the gear icon/’Points to path’ (3).
The Points to Path screen will appear/pop-up.
Use the ID (1) as order for point sequence. Then select an output location (2) and click run (3). When the process is finished click close (4).
Ok, that’s better. However, I want it more clear and headings. To do that we need to adjust the layer properties of the line layer and the point layer.
Double click on the line layer (1). Select the red glowing line (2) and adjust the size of the line to approximately 1,46 (3) and click ok (4).
Now double click on the point layer (1), select the simple marker (2) and change the icon to an arrow shape (3). In order to view the heading, each arrow has to be rotated by the same value of the heading of the ship. Click at the right side of the rotation menu on the icon (4) → ‘Field type: int, double, string’ (5) → click ‘Heading’ (6). To see the results click ok (7).
Results:
Final words
We can see that the middle part lacks a ‘few’ points. That can be caused by several reasons some legal and some illegal reasons. In this case, my computer rebooted for a mandatory update :(. But there are a few options to solve this lack of points partly. It involves the use of satellite imagery from the sentinel 1 constellation (SAR). If there is enough enthusiasm for, I will write about that the next time.
I’m not a professional programmer, so it is very likely that the code can be improved at some points. The code and the track are available on my github:
Program at:
https://gist.github.com/bearhunt11/9b02f4977436be2b1cf15c9b10a5feb0
CSV file at:
https://gist.github.com/bearhunt11/5ff44be9942a22e63f3317919564c44c
