- Sparse matrices can
be
represented efficiently
by using
multiple arrays
- There are 3 commonly used techniques:
- Coordinate-wise
- Compressed Sparse Row (CSR)
- Compressed Sparse Column (CSC) (not discussed)
- A sparse matrix is stored using 3 arrays:
- Val[N]:
contains the value of the non-zero elements
- Row[N]:
contains the row-index of the non-zero elements
- Col[N]: contains the column -index of the non-zero elements
- Val[N]:
contains the value of the non-zero elements
- For each non-zero element
in the matrix, there is
one entry
in
Val[], Row[] and Col[]:
- Row[x] = i
- Col[x] = j
- Val[x] = a
If A[i][j] = a , then for some index x: - Example:
0 1 2 3 4 5 6 7 0 [ 11 12 14 ] 1 [ 22 23 25 ] 2 [ 31 33 34 ] 3 [ 42 45 46 ] 4 [ 55 ] 5 [ 65 66 67 ] 6 [ 75 77 78 ] 7 [ 87 88 ]Representation using Coordinate-wise method: Index 0 1 2 3 4 5 6 7 8 9 10 ----------------------------------------------------------- Val 11 12 14 22 23 25 31 33 34 42 45 Row 0 0 0 1 1 1 2 2 2 3 3 Col 0 1 3 1 2 4 0 2 3 1 4 Index 11 12 13 14 15 16 17 18 19 20 21 ------------------------------------------------------------ Val 46 55 65 66 67 75 77 78 87 88 - Row 3 4 5 5 5 6 6 6 7 7 - Col 5 4 4 5 6 4 6 7 6 7 -
- Properties of the Coordinate-wise method:
- Rows or columns
need not be ordered
in any way.
- Storage requirement:
- nnz floating point, and
- 2*nnz integer (indices).
(nnz = Number of Non-Zero values)
- Main advantage
-
Accessing the matrix elements by
row or column
is
equally easy.
- Rows or columns
need not be ordered
in any way.
- Matrix-vector multiplication:
0 1 2 3 4 5 6 7 0 [ 11 12 14 ] [ 3 ] --> [11*3 + 12*5 + 14*1] 1 [ 22 23 25 ] [ 5 ] --> [22*5 + 23*2 + 25*0] 2 [ 31 33 34 ] [ 2 ] 3 [ 42 45 46 ] [ 1 ] 4 [ 55 ] x [ 0 ] = ??? 5 [ 65 66 67 ] [ 1 ] 6 [ 75 77 78 ] [ 2 ] 7 [ 87 88 ] [ 4 ] A[8][8] d[8]Index 0 1 2 3 4 5 6 7 8 9 10 ----------------------------------------------------------- Val 11 12 14 22 23 25 31 33 34 42 45 Row 0 0 0 1 1 1 2 2 2 3 3 Col 0 1 3 1 2 4 0 2 3 1 4 Index 11 12 13 14 15 16 17 18 19 20 21 ------------------------------------------------------------ Val 46 55 65 66 67 75 77 78 87 88 - Row 3 4 5 5 5 6 6 6 7 7 - Col 5 4 4 5 6 4 6 7 6 7 -
How is the result obtained: out[0] is computed as: [11*3 + 12*5 + 14*1] = A[0][0]*d[0] + A[0][1]*d[1] + A[0][3]*d[3] ^ ^ ^ | | | Val[0] Val[1] Val[2] = Val[0]*d[??] + Val[1]*d[??] + Val[2]*d[??] Look carefully at: Col[0], Col[1] and Col[2], you will see that: = Val[0]*d[Col[0]] + Val[1]*d[Col[1]] + Val[2]*d[Col[2]]
How can we tell that these value should be added to out[0] ??? Look at out[1]: [22*5 + 23*2 + 25*0] = A[1][1]*d[1] + A[1][2]*d[2] + A[1][4]*d[4] = Val[3]*d[1] + Val[4]*d[2] + Val[5]*d[4] = Val[3]*d[Col[3]] + Val[4]*d[Col[4]] + Val[5]*d[Col[5]] The value of Col[3], Col[4] and Col[5] are all 1 !!! In other words: Col[] tells you WHICH out[] element to use !
- Matrix-vector multiplication algorithm using
Coordinate-wise storage method
// Val[] = value array // Row[] = row index array // Col[] = column index array // nnz = # non-zero elements // m = # rows int k; // Clear out result for (k = 0; k < m; k = k + 1) { out[k] = 0.0; } // Matrix-vetor Multiply for (k = 0; k < nnz; k = k + 1) { out[Row[k]] = out[Row[k]] + Val[k]*in[Col[k]]; }
- Input parameters:
- double Val[]
- int Row[]
- int Col[]
- int d[]
- int m = # rows
- int nnz = # non-zero elements
- Output value(s):
- double out[]
- Matrix-vector multiple algorithm
void MatrixVectorMult(double out[], double Val[], int Row[], int Col[], double d[], int m, int nnz) { int k; for (k = 0; k < m; k = k + 1) { out[k] = 0.0; } for (k = 0; k < nnz; k = k + 1) { out[Row[k]] = out[Row[k]] + Val[k]*d[Col[k]]; } }
- How to use the algorithm
int main(int argc, char ** argv) { double Val[] = {11, 12, 14, 22, 23, 25, 31, 33, 34, 42, 45, 46, 55, 65, 66, 67, 75, 77, 78, 87, 88}; int Row[] = {0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 5, 5, 5, 6, 6, 6, 7, 7}; int Col[] = {0, 1, 3, 1, 2, 4, 0, 2, 3, 1, 4, 5, 4, 4, 5, 6, 4, 6, 7, 6, 7}; double u[8], v[8]; int i, j; u[0] = 3; u[1] = 5; u[2] = 2; u[3] = 1; u[4] = 0; u[5] = 1; u[6] = 2; u[7] = 4; MatrixVectorMult(v, Val, Row, Col, u, 8, 21); }
- Example Program:
(Demo above code)
- The Matrix-vector multiply function: click here
- A main program using the Matrix-vector multiply function: click here
- A sparse matrix is stored in
CSR format
also uses 3 arrays:
- Val[N]:
contains the
value of the non-zero elements
- RowPtr[N]:
contains the
row-index range
of the non-zero elements
- Col[N]: contains the column-index of the non-zero elements
- Val[N]:
contains the
value of the non-zero elements
- Example:
- The range
of
indices
are encoded as:
- [RowPtr[0] .. RowPtr[1])
- [RowPtr[1] .. RowPtr[2])
- [RowPtr[2] .. RowPtr[3])
- [RowPtr[3] .. RowPtr[4])
- [RowPtr[4] .. RowPtr[5])
- [RowPtr[5] .. RowPtr[6])
- [RowPtr[6] .. RowPtr[7])
- [RowPtr[7] .. RowPtr[8])
[ 11 12 14 ] [ 22 23 25 ] [ 31 33 34 ] [ 42 45 46 ] [ 55 ] [ 65 66 67 ] [ 75 77 78 ] [ 87 88 ]Representation using Compressed Sparse Row method: [0, 3) +-----------+ | | Index 0 1 2 3 4 5 6 7 8 9 10 ----------------------------------------------------------- Val 11 12 14 22 23 25 31 33 34 42 45 RowPtr 0 3 6 9 12 13 16 19 21 - - Col 0 1 3 1 2 4 0 2 3 1 4 Index 11 12 13 14 15 16 17 18 19 20 21 ------------------------------------------------------------ Val 46 55 65 66 67 75 77 78 87 88 - RowPtr - - - - - - - - - - - Col 5 4 4 5 6 4 6 7 6 7 -
- The range
of
indices
are encoded as:
- Properties of the Compressed Sparse Row method:
- Rows are stored
consecutively:
- Elements of row 0 are stored first
- Elements of row 1 are stored next
- And so on
- Columns need not be ordered in any way.
I.e.: elements within a row can be stored in any order
- Main advantage/disadvantage
- Storage required: nnz floating point, nnz+N+1 integer.
(nnz = Number of Non-Zero values, N = # rows of matrix)
(Require less storage than Coordinate-wise format) - Runs
faster
than Coordinate-wise representation
- Row access is easy, but
- column access difficult.
- Storage required: nnz floating point, nnz+N+1 integer.
- Rows are stored
consecutively:
- Matrix-vector multiplication:
0 1 2 3 4 5 6 7 0 [ 11 12 14 ] [ 3 ] --> [11*3 + 12*5 + 14*1] 1 [ 22 23 25 ] [ 5 ] --> [22*5 + 23*2 + 25*0] 2 [ 31 33 34 ] [ 2 ] 3 [ 42 45 46 ] [ 1 ] 4 [ 55 ] x [ 0 ] = ??? 5 [ 65 66 67 ] [ 1 ] 6 [ 75 77 78 ] [ 2 ] 7 [ 87 88 ] [ 4 ] A[8][8] d[8][0, 3) +-----------+ | | Index 0 1 2 3 4 5 6 7 8 9 10 ----------------------------------------------------------- Val 11 12 14 22 23 25 31 33 34 42 45 RowPtr 0 3 6 9 12 13 16 19 21 - - Col 0 1 3 1 2 4 0 2 3 1 4 Index 11 12 13 14 15 16 17 18 19 20 21 ------------------------------------------------------------ Val 46 55 65 66 67 75 77 78 87 88 - RowPtr - - - - - - - - - - - Col 5 4 4 5 6 4 6 7 6 7 -How is out[0] obtained: out[0] is computed as: [11*3 + 12*5 + 14*1] = A[0][0]*d[0] + A[0][1]*d[1] + A[0][3]*d[3] ^ ^ ^ | | | Val[0] Val[1] Val[2] = Val[0]*d[??] + Val[1]*d[??] + Val[2]*d[??] Look carefully at: Col[0], Col[1] and Col[2], you will see that: = Val[0]*d[Col[0]] + Val[1]*d[Col[1]] + Val[2]*d[Col[2]] How can we tell which number should be added to out[0] ??? ColPtr[0] = 0 ColPtr[1] = 3[3, 6) +-----------+ | | Index 0 1 2 3 4 5 6 7 8 9 10 ----------------------------------------------------------- Val 11 12 14 22 23 25 31 33 34 42 45 RowPtr 0 3 6 9 12 13 16 19 21 - - Col 0 1 3 1 2 4 0 2 3 1 4 Index 11 12 13 14 15 16 17 18 19 20 21 ------------------------------------------------------------ Val 46 55 65 66 67 75 77 78 87 88 - RowPtr - - - - - - - - - - - Col 5 4 4 5 6 4 6 7 6 7 -How is out[1] obtained: out[1] is computed as: [22*5 + 23*2 + 25*0] = A[1][1]*d[1] + A[1][2]*d[2] + A[1][4]*d[4] ^ ^ ^ | | | Val[3] Val[4] Val[5] = Val[0]*d[1] + Val[1]*d[2] + Val[2]*d[4] Look carefully at: Col[3], Col[4] and Col[5], you will see that: = Val[3]*d[Col[3]] + Val[4]*d[Col[4]] + Val[5]*d[Col[5]] How can we tell which number should be added to out[1] ??? ColPtr[1] = 3 ColPtr[2] = 6
- Matrix-vector multiplication algorithm using the
Compressed Sparse Row representation:
// Val[] = value array // RowPtr[] = row index RANGE array // Col[] = column index array // m = # rows int i, k; // Clear out result for (k = 0; k < m; k = k + 1) { out[k] = 0.0; } // Matrix-vetor Multiply for (i = 0; i < m; i = i + 1) { for (k = RowPtr[i]; k < RowPtr[i+1]; k = k + 1) { result[i] = result[i] + Val[k]*d[Col[k]]; } }
- Input parameters:
- double Val[]
- int RowPtr[]
- int Col[]
- int d[]
- int m = # rows
- Output value(s):
- double out[]
- Matrix-vector multiple function
void MatrixVectorMult(double out[], double Val[], int RowPtr[], int Col[], double in[], int m) { int i, k; for (k = 0; k < m; k = k + 1) { out[k] = 0.0; } for (i = 0; i < m; i = i + 1) { for (k = RowPtr[i]; k < RowPtr[i+1]; k = k + 1) { out[i] = out[i] + Val[k]*in[Col[k]]; } } }
- How to use the algorithm
int main(int argc, char ** argv) { double Val[] = {11, 12, 14, 22, 23, 25, 31, 33, 34, 42, 45, 46, 55, 65, 66, 67, 75, 77, 78, 87, 88}; int RowPtr[] = {0, 3, 6, 9, 12, 13, 16, 19, 21}; int Col[] = {0, 1, 3, 1, 2, 4, 0, 2, 3, 1, 4, 5, 4, 4, 5, 6, 4, 6, 7, 6, 7}; double u[8], v[8]; int i, j; u[0] = 3; u[1] = 5; u[2] = 2; u[3] = 1; u[4] = 0; u[5] = 1; u[6] = 2; u[7] = 4; MatrixVectorMult(v, Val, RowPtr, Col, u, 8); }
- Example Program:
(Demo CSR representation)
- The Matrix-vector multiply function: click here
- A main program using the Matrix-vector multiply function: click here
- The
most important advantage
of the CSR
representation is
increased speed
- Consider the
number of memory accesses
needed to perform the
matrix-vector multiplication
- Consider the statement
used to process a
Non-zero element
in the
Coordinate-wise method :
result[ Row[k] ] = result[ Row[k] ] + Val[ k ]*d[ Col[ k ] ];
Executing this statement requires the computer to obtain the following values from the main memory:
- Row[k] - (k is in a register in the CPU)
- Col[k]
- Val[k]
- d[(Col[k])] - You already have (Col[k])
- result[(Row[k])] - You already have (Col[k])
- Now, consider the statement
used to process a
Non-zero element
in the
CSR method :
result[ i ] = result[ i ] + Val[ k ]*d[ Col[k] ];Executing this statement requires the computer to obtain the following values from the main memory:
- Col[k] - (k is in a register in the CPU)
- Val[k]
- d[(Col[k])] - You already have (Col[k])
- result[i] - (i is in a register in the CPU)
- You perform
one less memory access
per non-zero element
Main memory access time is the bottle neck of computer systems...