Beginner's Reader in Parallel Computing

1 Introduction to Parallel Computing

An introduction to parallel computing must explain what a parallel program is and give a first impression how a computer system executes such programs. We need some terms to describe parallel programs as well as an approach how to develop it.

The reader in a great hurry finds a very condensed introduction in the tutorial Introduction to Parallel Computing from Blaise Barney of the Lawrence Livermore National Laboratory.

A more detailed view can be gained from the following readings.

• First overview
  Literature: chapter 1 from [RR10] (fulltext@KTH)

  • "Classical use of parallelism" (section 1.1, pp. 1-2)
    gives a brief motivation why we need parallel computing.
  • "Parallelism in today's hardware" (section 1.2, pp. 2-3)
    describes the need for parallel computing arising from recent trends in the hardware development.
  • "Basic concepts" (section 1.3, pp. 3-5)
    starts with the introduction of some terms needed all the time.

• Computer architecture
  Literature: chapter 2 from [RR10] (fulltext@KTH)

  • "Processor architecture and technology trends" (section 2.1, pp. 7-10)
    Modern computer systems support parallel computing at different levels that provide important contributions to the overall performance. A first overview is given here and later sections of this reader will provide hints how to use them efficiently.
  • "Flynn's taxonomy of parallel architectures" (section 2.2, pp. 10-12). For some nice illustrations see also Flynn's taxonomy in the Wikipedia.
    This broad classification from the 1960-ies has not a to big importance nowadays because many variations and mixed approaches exist. However, one should know it and will recognise it to a certain extend also in modern computer systems.
  • "Memory organisation of parallel computers" (section 2.3, pp. 12-17)
    Sections: 2.3.1 "Computers with distributed memory organization" and 2.3.2 "Computers with shared memory organization". The organisation of the memory systems has great influence on the performance of computations as well as on the possibilities to parallelise a program.

• Programming for parallel computing
  Main literature: chapter 3 from [RR10] (fulltext@KTH)

  • "Models for parallel systems" (section 3.1 pp. 93-95)
    provides a few basic terms that ease to talk about hard- and software.
  • "Parallelization of programs" (section 3.2 pp. 96-98)
    is a description of the general steps to create a parallel program.
  • "Levels of parallelism" (section 3.3 pp. 98-104)
    explains instruction level parallelism, data parallelism and loop parallelism.
  • A parallel programming model (section 1.3 of [Foster95])
    presents a task-oriented programming model, also sometimes called "functional programming model".
  • "Data distributions for arrays" (section 3.4, pp. 113-117)
    It is sufficient to get an impression of the common distributions from figure 3.7 on page 115 in the beginning. The more formal description in the text is useful in later programming projects.
  • "Information exchange" (section 3.5, pp. 117-125
    provides a conceptual view of communication in parallel programs.
  • "Parallel matrix-vector product" (section 3.6, pp. 125-130)
    gives a detailed example that demonstrates some of the previously introduced concepts.

2 Programming using message-passing

Blaise Barney's tutorial Message Passing Interface (MPI) provides a brief introduction to this programming paradigm.

(The information for more detailed studies is coming soon.)

3 Programming using shared-memory systems

Two other tutorials of Blaise Barney help to start programming with OpenMP resp. Pthreads.

(The information for more detailed studies is coming soon.)

4 Performance analysis of parallel programs

(The information for more detailed studies is coming soon.)

Appendix A: Literature