ScaleSFL: Scalable blockchain-based Sharding solution for Federated Learning

This project implements ScaleSFL (Scalable blockchain-based Sharding solution for Federated Learning). This approach extends previous approaches such as Committee Consensus, by providing a sharding mechanism, where model updates are verified independently within each shard, and further aggregated to produce a global model.

This implemention uses Hyperledger Fabric as the blockchain platform, using Fabric channels as independent "shards". Federated Learning is implemented using the Flower framework. This allows for further scalability, and capacity of the network. We test our approach using Hyperledger Caliper, a benchmarking tool.

Getting Started

Prerequisites

Make sure you have the proper Hyperledger Fabric prerequisites

To get the binaries required to run the project, you can run the command

curl -fsSL https://raw.githubusercontent.com/hyperledger/fabric/master/scripts/bootstrap.sh | bash -s

Next check the commands available by running

cd test-network
./network.sh -h

Running the network

This project relies on several independent componenets. The first step is to bring up the test network in order to use the Fabric blockchain. This is based on the test-network provided by fabric-samples, modified to handle many peers & organizations. Each peer will join a different channel (shard) by default.

Create Peer Image

We'll first need to create a fabric peer image with our endorsement plugin built in. Instructions to so can be found in the plugins folder.

Launch Peer Workers

The endorsement plugin relies on the peers for this network, we must launch the peer workers prior to launching the fabric network. We can run locally by using the manager script in example-participant/fl-service. You can find instructions in example-participant, please follow these and run the manager script, e.g.

python manager.py -p 8 -s 8 --num-threads=1

Launch Fabric network

Now we can bring up the network by running

./startFabric.sh

We can verify each of the peers have joined the correct channel by viewing the channels

Set the correct binaries path, and config directory

export PATH=$(realpath ${PWD}/../bin):$PATH
export FABRICCFGPATH=$(realpath ${PWD}/../config/)
. scripts/envVar.sh setGlobals 1

We can check which channels a peer is in using

peer channel list

Shutdown the network

Finally we can bring the network down using

./network.sh down

Testing

We perform workload tests using Hyperledger Caliper. Instructions to run this on the existing network can be found in caliper-tests.

The analysis for these tests can be found in the notebooks.

The main findings from these results show this solution scales with the number of shards