Studying how Different Hyper-parameters Affect Particle Swarm Optimization

by George Tambouratzis (Athena Research Centre) and Gary Pampara (Stellenbosh University University)

Computational Intelligence algorithms are designed to mimic the way groups of organisms collaborate to achieve a certain objective [1]. In computational intelligence there are several hyper-parameters that govern the performance of the algorithm and that must be properly set-up. Here we summarise the HYPCIA project, aimed at determining the best values for these hyper-parameters, in order to help researchers choose more appropriate values (at least as a starting point) to solve specific real-world tasks.

HYPCIA Project Definition 
One of the most widely used Computational Intelligence algorithms is Particle Swarm Optimisation (PSO), which is inspired by flocks of birds where individuals exchange information to attain their target [1]. PSO replicates a population of simple particles that search the pattern space to find an optimal solution. Each particle is characterized by its current location and its speed vector. The efficiency and accuracy with which this solution is found depends on the choice of specific hyperparameter values.

The project aims to determine how key PSO hyper-parameters need to be set up to obtain superior performance in  real-world tasks. By design, each PSO particle takes into account three types of information, (1) the best solution it has found so far, (2) the best solution found by all neighbouring particles and (3) particle velocity. These are adjusted by externally-defined weights, namely (1) c1 (cognitive coefficient), (2) c2 (social coefficient) and (3) w (inertia weight coefficient). These weights form a triplet of hyper-parameters (collectively termed hyper-parameter set) that largely governs the PSO convergence behaviour. In HYPCIA, the aim is to define the optimal values for the three coefficients, giving the highest and most consistent optimisation performance, across a range of swarm sizes and different benchmark functions.  

Methodology
he first part of the project consists of selecting potentially superior hyper-parameter sets, largely based on hyperparameter sets proposed in the literature. A set of established benchmark functions known to be hard-to-optimise are used to measure the ability of each hyper-parameter set to reach a good optimisation solution. PSO is applied to optimise these functions over different dimensionalities with a fixed budget of evaluations to test each configuration’s ability to reach the best solution with limited computational cost.

 All experiments are run with fifteen different swarm sizes, to test how each configuration performs with different swarm populations. The Cilib framework [2] is used to run experiments. Finally all configurations are repeated 25 times with different initialisations to eradicate random effects.

Then, to compare different hyperparameter sets, a pair-wise comparison is carried out, considering initially all pairs of sets. The steps to compare the hyper-parameter sets are summarized below:

• Step 1. Compare pairs of different parameter sets, for the same configuration (swarm size, benchmark function and function dimensionality), using rank-sum tests to rank the 25 runs corresponding to the first parameter set and 25 for the second parameter set.
• Step 2. Find out if the population of results for the first parameter set are statistically superior (or inferior) to the second parameter set by performing a statistical test (Wilcoxon Rank-sum test).
• Step 3. Then, aggregate results by counting statistically significant wins and losses across pairs of sets to identify how good one hyper-parameter set is against all others cumulatively.
• Step 4. Study how consistent the behaviour is across swarm sizes in terms of wins and losses.

An example is shown in Figure 1 comparing the hyper-parameter set proposed in [1] against thirteen other sets. The number of wins exceeds that of losses indicating that this hyperparameter set represents a good choice.

In choosing the best set of values, other factors may also be taken into account. For instance, is a function less consistent than another by being more effective for a specific range of sizes only? This would imply that it may be more difficult to fine-tune? Factors such as the gradient (in absolute values) of the curves of wins and losses and metrics (such as R2) need to be considered.

Figure 1: Comparison of wins and losses between the set of [1] and all other sets across different swarm sizes. Wins consistently exceed losses indicating a good hyper-parameter set.

Conclusions
The HYPCIA project aims to support the choice of the best hyper-parameter values for the PSO algorithm, by assimilating a large number of observations via a statistical-based approach. The project aims to determine for a general setup the best parameter values for PSO, to allow better performance to be achieved by development teams with limited experimentation in real-world applications. Detailed results will be reported in forthcoming publications and announcements. 

Acknowledgements
We acknowledge Prof. Andries Engelbrecht for his contribution to setting up the experiments. Most simulations have been implemented on the ARIS HPC architecture with support by GRNET via the HYPCIA project. 

References:
[1] J. Kennedy and R.C. Eberhart, “Particle Swarm Optimisation”, in Proc. of the IEEE Int. Conf. on Neural Networks, Perth, Australia, November 1995, pp. 1942-1947.
[2] G. Pampara and  A. P. Engelbrecht, 2015. “Towards a Generic Computational Intelligence Library: Preventing Insanity”, in Proc of the SSCI-2015 Symposium, pp. 1460-1469, 2015.

Please contact: 
George Tambouratzis, Athena Research Centre, Greece
This email address is being protected from spambots. You need JavaScript enabled to view it.