NoSync: Particle swarm inspired distributed DNN training

Mihailo Isakov, Michel A. Kinsy

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Training deep neural networks on big datasets remains a computational challenge. It can take hundreds of hours to perform and requires distributed computing systems to accelerate. Common distributed data-parallel approaches share a single model across multiple workers, train on different batches, aggregate gradients, and redistribute the new model. In this work, we propose NoSync, a particle swarm optimization inspired alternative where each worker trains a separate model, and applies pressure forcing models to converge. NoSync explores a greater portion of the parameter space and provides resilience to overfitting. It consistently offers higher accuracy compared to single workers, offers a linear speedup for smaller clusters, and is orthogonal to existing data-parallel approaches.

Original languageEnglish (US)
Title of host publicationArtificial Neural Networks and Machine Learning – ICANN 2018 - 27th International Conference on Artificial Neural Networks, 2018, Proceedings
EditorsYannis Manolopoulos, Barbara Hammer, Ilias Maglogiannis, Vera Kurkova, Lazaros Iliadis
PublisherSpringer Verlag
Number of pages13
ISBN (Print)9783030014209
StatePublished - 2018
Externally publishedYes
Event27th International Conference on Artificial Neural Networks, ICANN 2018 - Rhodes, Greece
Duration: Oct 4 2018Oct 7 2018

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11140 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference27th International Conference on Artificial Neural Networks, ICANN 2018


  • Artificial neural network
  • Deep learning
  • Distributed systems
  • Evolutionary algorithm
  • Particle swarm optimization

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)


Dive into the research topics of 'NoSync: Particle swarm inspired distributed DNN training'. Together they form a unique fingerprint.

Cite this