Computer Vision Talk

Computer Vision Talk

What the funk?


In my most recent visit to Mexico, a very dear friend (Rolis) of mine (Aldux) invited me to give a talk about computer vision and the applications I usually use it for, which are drones of course!

Needless to say, I had a blast!! I talked a lot of the computer vision slung load technique that I used in my PhD thesis as well as the cool project I developed as Postdoc at the University of Oxford, the Kingbee project!

Also, a week before the talk, I created some new scripts on my popular computer vision repository, those ones relate to haar cascades.

Slung load recreation with microphone

One of the most interesting new items is a script that detects cars from a webcam… such webcam is an open traffic IP cam somewhere in the USA… One of the most complex parts when doing this script was to be able to open correctly the stream of images, then the haar cascade is very easily implemented, I took the trained xml files from other repositories similar to mine (proper source crediting is given of course).

You can see it in action here:


Another cool script is one that detects people (the full body) again using as well haar cascades. This one comes from an IP camera in Spain, you can see it in action here:


The talk was given at my friends company, called Funktionell which is a pretty cool place! full of gadgets, electronics, 3d printers, engineers, programmers and designers!! Their statement is:

Somos una empresa de tecnología que se dedica a crear experiencias digitales inolvidables. Movidos por la innovación y la curiosidad por romper paradigmas, mezclamos la tecnología con la imaginación para llegar a resultados de calidad increíbles.

Finally, the video of the entire talk was posted on Funktionell facebook page, it can be seen here:

Visión por computadora

Bienvenidos al #CodeMeetsFunk
Hablamos de Visión por computadora aplicada en detección de rostros, colores y control de drones.
Ponentes: Gustavo Heras y el Dr. Aldo Vargas

Posted by Funktionell on Saturday, April 1, 2017


Slung Load Controller

Slung Load Controller

Multirotor Unmanned Aerial Vehicles (MRUAV) have become an increasingly interesting area of study in the past decade, becoming tools that allow for positive changes in today’s world. Not having an on-board pilot means that the MRUAV must contain advanced on- board autonomous capabilities and operate with varying degrees of autonomy. One of the most common applications for this type of aircraft is the transport of goods. Such applications require low-altitude flights with hovering and vertical take-off and landing (VTOL) capabilities.

Similar as before in this project we use the AltaX Flight Stack which is compromised by a Raspberry Pi 3 as companion computer and a naze32 as flight controller.

The slung load controller and the machine learning estimator is running on the RPI3, although of course the training of the recurrent neural network was done offline in a big desktop computer. The RPI calculates the next vehicle position based on the estimation of the position of the slung load, everything is running using our framework DronePilot and guess what? its open source ;). Keep reading for more details.

If the transported load is outside the MRUAV fuselage, it is usually carried beneath the vehicle attached with cables or ropes, this is commonly referred to as an under-slung load. Flying with a suspended load can be a very challenging and sometimes hazardous task because the suspended load significantly alters the flight characteristics of the MRUAV. This prominent pendulous oscillatory movement affects the response in the frequency range of the attitude control of the vehicle. Therefore, a fundamental understanding of the dynamics of slung loads as they relate to the vehicle handling is essential to develop safer automatic pilots to ensure the use of MRUAV in transporting load is feasible. The dynamics of the slung load coupled to a MRUAV are investigated applying Machine Learning techniques.

The learning algorithm selected in this work is the Artificial Neural Network (ANN), a ML algorithm that is inspired by the structure and functional aspects of biological neural networks. Recurrent Neural Network (RNN) is a class of ANN that represents a very powerful system identification generic tool, integrating both large dynamic memory and highly adaptable computational capabilities.

Recurrent neural network diagram

In this post the problem of a MRUAV flying with a slung load (SL) is addressed. Real flight data from the MRUAV/SL system is used as the experience that will allow a computer software to understand the dynamics of the slung in order to propose a swing-free controller that will dampen the oscillations of the slung load when the MRUAV is following a desired flight trajectory.

This is achieved through a two-step approach: First a slung load estimator capable of estimating the relative position of the suspension system. This system was designed using a machine learning recurrent neural network approach. The final step is the development of a feedback cascade control system that can be put on an existing unmanned autonomous multirotor and makes it capable of performing manoeuvres with a slung load without inducing residual oscillations.

Proposed control strategy

The machine learning estimator was designed using a recurrent neural network structure which was then trained in a supervised learning approach using real flight data of the MRUAV/SL system. This data was gathered using a motion capture facility and a software framework (DronePilot) which was created during the development of this work.

Estimator inputs-outputs

After the slung load estimator was trained, it was verified in subsequent flights to ensure its adequate performance. The machine learning slung load position estimator shows good performance and robustness when non-linearity is significant and varying tasks are given in the flight regime.

Estimator verification

Consequently, a control system was created and tested with the objective to remove the oscillations (swing-free) generated by the slung load during or at the end of transport. The control technique was verified and tested experimentally.

The overall control concept is a classical tri-cascaded scheme where the slung load controller generates a position reference based on the current vehicle position and the estimated slung load position. The outer loop controller generates references (attitude pseudo- commands) to the inner loop controller (the flight controller).

Control scheme

The performance of the control scheme was evaluated through flight testing and it was found that the control scheme is capable of yielding a significant reduction in slung load swing over the equivalent flight without the controller scheme.

The next figures show the performance when the vehicle is tracking a figure-of-eight trajectory without control and with control.

The control scheme is able to reduce the control effort of the position control due to efficient damping of the slung load. Hence, less energy is consumed and the available flight time increases.

Regarding power management, flying a MRUAV with a load will reduce the flight times because of two main factors. The first one relates to adding extra weight to the vehicle, consequently the rotors must generate more thrust to keep the desired height of the trajectory controller, hence reducing the flight time. The second factor relates to aggressive oscillations of the load for this reason. The position controller demands faster adjustment to the attitude controller which increases accordingly the trust generated by the rotors. The proposed swing-free controller increases the time of flight of the MRUAV when carrying a load by 38% in comparison with the same flight without swing-free control. This is done by reducing the aggressive oscillations created by the load.

The proposed approach is an important step towards developing the next generation of unmanned autonomous multirotor vehicles. The methods presented in this post enables a quadrotor to perform flight manoeuvres while performing swing-free trajectory tracking.

Don’t forget to watch the video, it is super fun:

UoG 360 Spherical

UoG 360 Spherical

Glasgow University area 360-degree spherical panoramic photo*.

ƒ/2.2 4.7 mm
1/120 ISO 151

* Complying w/ UK Air Navigation Order (CAP393). Always remember to fly safe!


Computer vision using GoPro and Raspberry Pi

Computer vision using GoPro and Raspberry Pi

In this post I’m going to demonstrate how to do test some computer vision techniques using the video feed from a GoPro Hero 3 directly towards a Raspberry Pi 3.

I’m using a special bridge that has a HDMI input and as an output goes to the CSI camera port of the Raspberry Pi, so, basically as easy as using a RPI camera…


This is actually not a very common technique to do, the bridge is from the company Auvidea, and the model is the B101.

And the best part is that its plug and play. I just install it on my RPI, connected the CSI cable to the camera port, turn my RPI on, turn the GoPro on, and run “raspivid -t 0” and voilà, you will see the video on the screen!!!

Different angle of the rpi + HDMI bridge
Different angle of the rpi + HDMI bridge

After that is just question of using my computer vision repository:, and start testing the different scripts… As usual, I created a video for you guys to see it working, take a look here:



Carbon fibre foldable quadcopter

BigX is a bespoke vehicle designed and built to support research projects. It’s big, with a 900mm wheelbase which means it can hold up to 21in propellers.


The frame is manufactured by SoliDrone, the model of the frame is FR4X 900F. I got this prototype frame in order to build it and test it. The company will start selling this great frame soon, so, check their website for updates. They have a beautiful render that you can see here:



Specifications of the vehicle:

Frame: FR4X 900F
Motors: Foxtech S5010 288kv
Propellers: 18×6.5in CF
Wheebase: 900mm
FC: Pixhack
ESCs: Hobbywing XRotor Pro 40A
Weight(no batt): 3kg


The carbon fibre plates are really thick, 3mm, which makes it very very hard and solid… SoliDrone, hehe… But that is one of the reasons why is a bit heavy. But considering the type of applications this one is going to be use in, it is just right. It’s big, 900mm wheelbase, and because of that, it can be equipped with very large propellers, but being foldable, makes it very easy to transport. This is a great feature.


Building process:


So, how big it is??


Weight with 16,000mah battery
Weight with 16,000mah battery

At the moment I’m putting 18in props to this one, and its performing quite well, maybe I will put bigger props on the future.

Hovering time:

I did several tests using two different batteries. Both  batteries are Multistar LiHV from HobbyKing. The longest flight was almost 32 minutes.

Battery Flight Time
10,000 mah 25 min
16,000 mah 32 min

Then I added a Raspberry Pi and using DronePilot, I made it fly autonomously in very different ways, and it performs great!! You can see in the video how good it flies. And as usual, the Scottish weather does not help, but this vehicle was able to fly under raining conditions and strong wind gusts, with ease.



Trajectory Controller

Trajectory Controller

A while ago, we reviewed how a hover controller works, in this post we are going to discuss how to go a bit further and create a trajectory controller.

In that previous blog post we discussed about how to control a drone to make it hold a specified position. This refers to the part of Control in the GNC argo. This refers to the manipulation of the forces, by way of steering controls, thrusters, etc., needed to track guidance commands while maintaining vehicle stability.

In this part we are focused on the Guidance. This refers to the determination of the desired path of travel (the “trajectory”) from the vehicle’s current location to a designated the target, as well as desired changes in velocity, rotation and acceleration for following that path.

There is several steps before we can achieve this. Mainly the next ones:

  1. Fly the vehicle using the flight stack
  2. Design a controller that will track/hold a specified position
  3. Create a trajectory, based on time and other factors

For the first part, in this blog we will use Altax Flight Stack, that compromises a companion computer and a flight controller. In this particular case I’m using a naze32 as flight controller, and two companion computers: Raspberry Pi 2 and oDroid U3.

The naze32 is connected to the Odroid U3 via a usb cable (a very short one). The vehicle is a 330mm rotor to rotor fiber glass frame, with 7×3.8in propellers, 1130kv motors, 15amps ESCs and a 3000mah 10C battery. It will fly for 11-13 minutes.

The Odroid U3 is running Ubuntu 14.04.1 in a eMMC module, which makes it boot and run generally faster. Its being powered by a BEC that is connected to the main battery.

The companion computer will “talk” a special language in order to send commands to the vehicle, this one is described here. And the most important part is that it will run the DronePilot framework. This framework is the that will pilot the vehicle.

For the second part (position controller), you can refer to this page to see how it works.

And now the trajectory part…

We need to generate certain X and Y coordinates that then it will be “fed” to the position controller at a specific time. We are going to create two types of trajectories, circle and a infinity symbol. Why this ones? because this ones are easy to generate and perfect to excite all the multi-rotor modes.

How to generate a circle trajectory??


This one is very simple… there is basically two parameters needed… Radius and angle. In our case the angle part we are going to combine it with the step time of the main loop of the controller and pi… basically the angle one will go from 0 to 360 degrees (in radians of course). The code looks like this:


So, if we declare “w” like this: (2*pi)/12 it means that the trajectory will take 12 seconds to complete a full revolution, and then start over. This is the parameter that we will change if we want the vehicle to travel faster. Its better to start with a slow value, and then progress to faster trajectories.

The next step is to “fed” this coordinates to the position controller inside the control loop. That is done in this script.

The infinity trajectory is a special one! this one is called in several ways: Inifity trajectory, figure of eight… And there is several ways of how to calculate the coordinates, you can see in the next gif the posibilites of how to create a figure of eight:


The one I like the red dot one! why is this?? that one is called the Lemniscate of Bernoulli, which is constructed as a plane curve defined from two given points F1 and F2, known as foci, at distance 2a from each other as the locus of points P so that PF1·PF2 = a2.


This lemniscate was first described in 1694 by Jakob Bernoulli as a modification of an ellipse, which is the locus of points for which the sum of the distances to each of two fixed focal points is a constant. We can calculate it as a parametric equation:


And then the rest is feeding that information to the position controller which will try to follow that trajectory as the dots on the plots. Magic.



The cool video can be seeing here: