Map Reduce System

Requirements from the binaries

All binaries must be statically linked.

Map

Forms a list of (key, value) pairs and saves them to the output file, the exact format of key value pairs must be consistent with other binaries.

./map <input_file> <output_file>

Combiner

Provides intermediate reduce after the map phase, we encourage to use the same binary as reduce.

./combiner <input_file> <output_file>

Partitioner

Partitions the key value pairs into R files stored in the output directory, it must partition it so that the file names are sorted by partition.

./partitioner <R> <input_file> <output_dir>

Reduce

Reduces all the files in the input directory to one output file.

./reduce <input_dir> <output_file>

Deployment

Login to Google Cloud

gcloud auth login

Set up kubernetes cluster

Set up kubernetes cluster and get credentials.

gcloud container clusters create-auto mr-cluster --region us-central1
gcloud container clusters get-credentials mr-cluster --region us-central1

Set up artifact registry for docker images

gcloud artifacts repositories create mr-repository --repository-format=docker \
    --location="us-central1" --description="Docker repository" \
    --project=$GOOGLE_CLOUD_PROJECT

Build images

bash bulild_docker.sh

Deploy images

bash deploy.sh

Get Master's external IP

kubectl get svc

Client usage

In the client directory run:

 mvn -e exec:java -Dexec.mainClass="mimuw.mapreduce.Client" -Dexec.args="<host> <port>"

then follow the instructions from the client.