SARUAV - POLISH DRONE SOFTWARE FOR SAVING LIVES

Today we present one of the newest products in our offer, the SARUAV software, which facilitates and accelerates search and rescue operations with the use of drones. Although this project, created at the University of Wroclaw, has only recently entered the commercial stage, it can already boast of the first successes! Bieszczady GOPR (Polish Tatra Mountains Volunteer Rescue Service) has used this system to search for a missing person and the person was found thanks to SARUAV. You can find more information on this subject in this video.

SIGN OF THE TIMES

Drones have always gone hand in hand with the latest technology - no wonder, after all, if it wasn't for modern, miniaturized microprocessors, sensors, or satellite navigation, the world of unmanned aerial vehicles would look completely different. With the miniaturization of digital cameras over the past few years we have seen better and better sensors and optics mounted on gimbals, later also in obstacle sensors and more... One of the latest technologies that is slowly seeping into our lives at various levels is automatic image analysis based on artificial intelligence. Facial recognition by a camera or phone no longer surprises anyone, but this is just the tip of the iceberg. A very interesting example of a leading drone solution in this area is the Aerialtronics Pensar, an intelligent camera available with Altura Zenith drones and some Dronevolt drones. What is its uniqueness based on? Thanks to the on-board NVIDIA JETSON mini-supercomputer, the camera continuously evaluates and classifies what it is looking at, of course relaying this information to the operator. With the flexibility of this technology, such a camera can literally be taught to recognize all sorts of objects - including recognizing the face of a specific person, listing serial numbers of hard-to-reach devices, identifying rust on connectors, etc. etc.

A.I., MACHINE LEARNING, NEURAL NETWORKS? WHAT'S ALL THAT...?

 

One of the most common technologies in AI (Artificial Intelligence) are the so-called neural networks. This is a type of algorithm or program, which in its pattern of operation resembles a neuron. In other words, it has several inputs, which are used to receive information, "inference module", which generates an output signal on the basis of input information and their weights. Then the processed information is directed to the output and passed on. In this way, it is possible to compare, for example, an image seen by a camera with a previously created pattern. Furthermore, these patterns can be developed and machines can be taught new patterns through a machine learning process. Basically, it boils down to the fact that we show our algorithm as many selected images as possible, e.g. depicting faces, and tell it to create itself a class of objects with facial features based on that. The more and the better data we use for learning, the better algorithm will detect these faces. We can do the same with the face of a particular person (i.e. the machine will distinguish between John Smith and Adam Smith) or actually anything else. Interestingly, there are already companies that teach existing AI systems or sell databases of photos (and other media) representing specific classes of objects for learning intelligent systems. Another application of AI is to recognize the manufacturer and model of a drone by its radio signal - such solutions are used by some anti-drone systems.

SUCCESS OF THE UNIVERSITY OF WROCLAW AND SARUAV COMPANY

 

A research and development group at the University of Wroclaw has decided to combine artificial intelligence with aerotriangulation known from photogrammetry to create a program that will revolutionize search and rescue operations. We must admit that the product that resulted from this cooperation is a really interesting, complete and useful tool. The team approached the task with academic meticulousness, at the same time being perfectly aware that it must be a practical and easy to use tool. This goal was achieved in 100%. During the development process, the team cooperated closely with both manufacturers of unmanned aerial vehicles and search and rescue groups, including the Jurajska GOPR Group, Anna Pasek Foundation, Search and Rescue Group of Voluntary Fire Brigade Jarogniewice, Special Rescue Unit of Voluntary Fire Brigade Wrocław, and many others. Thanks to that, the scientific project turned into a ready commercial product, which has been repeatedly tested in the field.

SOUNDS FANTASTIC, BUT HOW DOES IT WORK IN PRACTICE?

 

Practice is the first thing the developers had in mind when creating SARUAV. The team from Wrocław University (consisting of Prof. Tomasz Niedzielski, Dr. Mirosława Jurecka and Dr. Bartłomiej Miziński) focused on ensuring that the programme was easy to use and had a clear, intuitive workflow. The point is that the interface of the program should be easy enough to use it without problems and without mistakes even in very stressful situations, under great time pressure. And indeed - we learned how to use the program almost immediately (just before the first presentation ;) ). Okay, but let's get down to specifics.

The program consists of two modules: determination of the search area and analysis of aerial photos. In the first module we enter the data: the last place and time when the searched person was seen, and the place to which the person was going (if we know it). Then we give the radii of the circles and the time of the walk. What radii? Most search and rescue units have data for their area regarding how far (within what radius) a particular type of person (child, adult, elderly, etc.) can travel in their area. If the local unit does not have such statistics, you can use existing scientific studies, or ask the manufacturer what values would be optimal. It is also possible to determine them "by eye" - the program will still produce useful data on this basis in the form of a color-coded mobility map, which itself takes into account the field data. This is the reason why the program is licensed for a specific area: because the maps of terrain "resistance" are installed together with the program, i.e. how easy/difficult it is for a pedestrian to move in a given area. This takes into account roads and paths (which speed up movement), type of terrain (sloping, flat, wooded, swampy, etc.), and terrain obstacles (lakes, rivers, etc.).

Such a map is obviously useful in planning a search raid, but it can also help in organizing the ground search part. What's more, we can add various additional layers to the map, for example, paths of the rescuers' passages recorded from handheld GPS devices (e.g. in GPX format).

Then we move to the determination of the raid area. This is done in software specific to the drone model (e.g. DataPilot in the case of H520/H520E). We need to use a drone with a compatible camera with the highest possible resolution, such as the Yuneec E90 or E90x. Most other popular brands such as DJI or Autel are also supported. The list is constantly extended and thanks to the fact that the development team is from Poland and reacts very quickly, there is no problem to add a new camera to the list even within a few days. For this it is enough for the producer to have pictures from the new camera, on the basis of which its optical parameters will be calculated, which are needed by the algorithms to triangulate the position of the detected object. When planning a raid, one should follow the manufacturer's guidelines: it has to be a survey-type raid, with the camera pointed vertically downwards, with 70/70 coverage (the pictures have to overlap 70%), with terrain pixel size of 3cm, recommended speed not higher than 5 m/s (to minimize moving pictures). With the H520/E, all these parameters are easily entered in the flight plan in the app or on the PC, and planning a route takes no more than a minute.

Now the drone makes the flight and returns to the operator. While replacing the battery, we quickly remove the card from it, download the photos to the SARUAV operator's computer, and then send the drone out for another flight. The images are ready for analysis immediately after copying to disk. Here we move to the second module, which is the analysis of the photos. Simply point to the folder with the photos and the place where the drone starts and click start analysis. The program will analyse all the photos, immediately showing the possible hits, i.e. you do not have to wait until the program finishes the analysis to start evaluating the results. At this stage the SARUAV operator in the verification panel (in the picture on the right) evaluates whether it is a hit that we are interested in or a so-called false positive. This is because the program is "tuned" in such a way as not to miss any person for sure - therefore, there are also detections of objects similar to people. This is intentional, because it takes literally a second and one click to verify a photo in the panel. Photos can be marked as a hit (location to be verified by ground units), a miss (not human, or bystanders), or a question mark for further verification. The location of each hit is marked in the program with a pin on the map. A great and very useful feature is the generation of a report, which includes a location map, and the coordinates and photos of each hit and question mark. Such a report is generated as an html file, so it is lightweight and can be opened on almost any device.

Feel free to download the GOPR Bieszczady search raid report from the real (and successful) action I wrote about in the first paragraph.

 

As I think, there are similar doubts in your heads as in mine when I heard about this program. Is ripping images from the card to the computer really necessary? Why can't you work on the images sent by the drone to the camera? Well, first of all, we need the maximum resolution possible. The finesse of SARUAV lies in the fact that the program detects people who, because of the height of flight, occupy a small fraction of the frame and it is usually impossible for the operator to see such details, especially since the resulting live image is at best FHD resolution. Add to that fatigue, pressure, stress - it is easy to miss something. Meanwhile, the program is able to detect a partially obscured human body sticking out from under a bush somewhere on the edges of the frame. In addition, a very important aspect of this application is how easy it is to adapt it to new drones and cameras - just a few photos and a signal to the manufacturer. Emergency services do not always use their own drones - sometimes they are helped by various volunteers, other units or foundations. So the lack of integration with the flying platform is an advantage in the sense that it significantly facilitates and accelerates the expansion of the list of compatible cameras. The goal is to make the tool as widely available as possible in rescue operations. How quickly the development team reacts to market demands can be proved by our experience with water detection - here the algorithms used were not doing so well (relatively many false positives on buoys, boat engines), but after just a few days the team developed a new model of water detection, which solves these problems. If all manufacturers worked like this, life would be much simpler ;).

The analysis itself takes different amount of time, depending on the parameters of the computer - we recommend using gaming computers, mainly because of the larger RAM and efficient graphics card. However, these are not cosmic requirements, and a computer that meets the minimum requirements and will analyze the whole raid in a few minutes costs about 4-5 thousand zlotys.

However, we could convince about the advantages of this program at will - but the real significance is the opinion of those who are on the "front line" of the fight for human life - that is firefighters, rescuers and other services. And they speak unequivocally: this is a great, very useful tool, which can be crucial in many actions. Of course, we should be aware of the fact that SARUAV will not work everywhere - for example, where there are always a lot of bystanders (e.g. in cities) we will have too many positive hits. Forests with dense tree crowns are also a problem - but this is a logical limitation for any aerial search method. Nevertheless, tests in sparser forests (or when there is no foliage) are also very positive.

We are pleased to hear that in our country such modern and useful tools are created by people with a mission. It is even nicer to be able to distribute this product to the widest possible audience. If you have any questions about this system, feel free to contact us, we will try to help and provide comprehensive answers.