High Performance Computing - Charles Severance [116]
* Begin running the time steps
DO TICK=1,MAXTIME
* Set the heat persistent sources
CALL STORE(BLACK,ROWS,COLS,OFFSET,MYLEN,
+ ROWS/3,TOTCOLS/3,10.0,INUM)
CALL STORE(BLACK,ROWS,COLS,OFFSET,MYLEN,
+ 2*ROWS/3,TOTCOLS/3,20.0,INUM)
CALL STORE(BLACK,ROWS,COLS,OFFSET,MYLEN,
+ ROWS/3,2*TOTCOLS/3,-20.0,INUM)
CALL STORE(BLACK,ROWS,COLS,OFFSET,MYLEN,
+ 2*ROWS/3,2*TOTCOLS/3,20.0,INUM)
Now we perform the exchange of the “ghost values” with our neighboring processes. For example, Process 0 contains the elements for global column 50. To compute the next time step values for column 50, we need column 51, which is stored in Process 1. Similarly, before Process 1 can compute the new values for column 51, it needs Process 0’s values for column 50.
Figure 4.9 shows how the data is transferred between processors. Each process sends its leftmost column to the left and its rightmost column to the right. Because the first and last processes border unchanging boundary values on the left and right respectively, this is not necessary for columns one and 200. If all is done properly, each process can receive its ghost values from their left and right neighbors.
Figure 4.9. Pattern of communication for ghost values
The net result of all of the transfers is that for each space that must be computed, it’s surrounded by one layer of either boundary values or ghost values from the right or left neighbors:
* Send left and right
IF ( .NOT. IAMFIRST ) THEN
CALL PVMFINITSEND(PVMDEFAULT,TRUE)
CALL PVMFPACK( REAL8, BLACK(1,1), ROWS, 1, INFO )
CALL PVMFSEND( TIDS(INUM-1), 1, INFO )
ENDIF
IF ( .NOT. IAMLAST ) THEN
CALL PVMFINITSEND(PVMDEFAULT,TRUE)
CALL PVMFPACK( REAL8, BLACK(1,MYLEN), ROWS, 1, INFO )
CALL PVMFSEND( TIDS(INUM+1), 2, INFO )
ENDIF
* Receive right, then left
IF ( .NOT. IAMLAST ) THEN
CALL PVMFRECV( TIDS(INUM+1), 1, BUFID )
CALL PVMFUNPACK ( REAL8, BLACK(1,MYLEN+1), ROWS, 1, INFO
ENDIF
IF ( .NOT. IAMFIRST ) THEN
CALL PVMFRECV( TIDS(INUM-1), 2, BUFID )
CALL PVMFUNPACK ( REAL8, BLACK(1,0), ROWS, 1, INFO)
ENDIF
This next segment is the easy part. All the appropriate ghost values are in place, so we must simply perform the computation in our subspace. At the end, we copy back from the RED to the BLACK array; in a real simulation, we would perform two time steps, one from BLACK to RED and the other from RED to BLACK, to save this extra copy:
* Perform the flow
DO C=1,MYLEN
DO R=1,ROWS
RED(R,C) = ( BLACK(R,C) +
+ BLACK(R,C-1) + BLACK(R-1,C) +
+ BLACK(R+1,C) + BLACK(R,C+1) ) / 5.0
ENDDO
ENDDO
* Copy back - Normally we would do a red and black version of the loop
DO C=1,MYLEN
DO R=1,ROWS
BLACK(R,C) = RED(R,C)
ENDDO
ENDDO
ENDDO
Now we find the center cell and send to the master process (if necessary) so it can be printed out. We also dump out the data into files for debugging or later visualization of the results. Each file is made unique by appending the instance number to the filename. Then the program terminates:
CALL SENDCELL(RED,ROWS,COLS,OFFSET,MYLEN,INUM,TIDS(0),
+ ROWS/2,TOTCOLS/2)
* Dump out data for verification
IF ( ROWS .LE. 20 ) THEN
FNAME = ’/tmp/pheatout.’ // CHAR(ICHAR(’0’)+INUM)
OPEN(UNIT=9,NAME=FNAME,FORM=’formatted’)
DO C=1,MYLEN
WRITE(9,100)(BLACK(R,C),R=1,ROWS)
100 FORMAT(20F12.6)
ENDDO
CLOSE(UNIT=9)
ENDIF
* Lets all go together
CALL PVMFBARRIER( ’pheat’, NPROC, INFO )
CALL PVMFEXIT( INFO )
END
The SENDCELL routine finds a particular cell and prints it out on the master process. This routine is called in an SPMD style: all the processes enter this routine although all not at precisely the same time. Depending on the INUM and the cell that we are looking for, each process may do something different.
If the cell in question is in the master process, and we are the master process, print it out. All other processes do nothing. If the cell in question is stored in another process, the process