Distributed computing is a method of dividing computational tasks across multiple machines to achieve faster processing, higher availability, and better scalability. In the context of Linux, distributed computing can be implemented using various tools and frameworks such as Apache Hadoop, Apache Spark, and MPI (Message Passing Interface). This article will guide you through setting up a distributed computing environment on Linux, highlighting its importance and providing practical examples.

Examples:

1. Setting Up Apache Hadoop:

   Apache Hadoop is a popular framework for distributed storage and processing of large data sets.

   • Install Java:

     sudo apt update
     sudo apt install openjdk-11-jdk -y

   • Download and Install Hadoop:

     wget https://downloads.apache.org/hadoop/common/hadoop-3.3.0/hadoop-3.3.0.tar.gz
     tar -xzvf hadoop-3.3.0.tar.gz
     sudo mv hadoop-3.3.0 /usr/local/hadoop

   • Configure Hadoop Environment Variables: Edit ~/.bashrc and add the following lines:

     export HADOOP_HOME=/usr/local/hadoop
     export PATH=$PATH:$HADOOP_HOME/bin
     export JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

   • Configure Hadoop: Edit the configuration files located in $HADOOP_HOME/etc/hadoop/. For example, edit core-site.xml to set the default filesystem:

     <configuration>
      <property>
          <name>fs.defaultFS</name>
          <value>hdfs://localhost:9000</value>
      </property>
     </configuration>

   • Start Hadoop:

     $HADOOP_HOME/sbin/start-dfs.sh
     $HADOOP_HOME/sbin/start-yarn.sh

2. Setting Up Apache Spark:

   Apache Spark is another powerful tool for distributed data processing.

   • Download and Install Spark:

     wget https://downloads.apache.org/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
     tar -xzvf spark-3.1.2-bin-hadoop3.2.tgz
     sudo mv spark-3.1.2-bin-hadoop3.2 /usr/local/spark

   • Configure Spark Environment Variables: Edit ~/.bashrc and add the following lines:

     export SPARK_HOME=/usr/local/spark
     export PATH=$PATH:$SPARK_HOME/bin

   • Run Spark Shell:

     $SPARK_HOME/bin/spark-shell

3. Using MPI for Distributed Computing:

   MPI is a standard for parallel programming in distributed computing environments.

   • Install MPI:

     sudo apt update
     sudo apt install mpich -y

   • Write an MPI Program: Create a file named hello_mpi.c with the following content:

     #include <mpi.h>
     #include <stdio.h>
     
     int main(int argc, char** argv) {
      MPI_Init(NULL, NULL);
      int world_size;
      MPI_Comm_size(MPI_COMM_WORLD, &world_size);
      int world_rank;
      MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
      printf("Hello from rank %d out of %d processors\n", world_rank, world_size);
      MPI_Finalize();
      return 0;
     }

   • Compile and Run the MPI Program:

     mpicc -o hello_mpi hello_mpi.c
     mpirun -np 4 ./hello_mpi