GoldenGate Process Memory Usage

From the GG Installation documentation:

Each Extract and Replicat process needs approximately 25-55 MB of memory, or more depending on the size of the transactions and the number of concurrent transactions.

So let’s see some example of how much memory is being used. I wrote a short script to show the memory usage by using “ps” and have two examples, one on Linux and one on Solaris.

The first example ( on Linux ) shows the usage of GoldenGate extract and replicat processes configured to process a heartbeat table, so not very much activity.

The second example ( on Solaris ) show the usage of GoldenGate processes on a live system with lots of activity.

Due to differences on how ps behaves between Linux and Solaris, I display either the group name itself, or the parameter file of the group, which contains the group name. The unix script can be found at the bottom of this blog.

Here’s the example running on Linux. This configuration does not have very many transactions.


> ./gg_memory_usage.sh
OSNAME = Linux

#####################################
# Individual extract Process Usage #
#####################################
48.3516 MB CGGMONX
48.6094 MB CGGMONY
31.6055 MB PGGMONX
31.6055 MB PGGMONY

#####################################
#   Total extract Process Usage    #
#####################################
Number of processes      = 4
AVG Memory usage/process = 40.043 MB
Total memory usage       = 160.172  MB

#####################################
# Individual replicat Process Usage #
#####################################
29.8398 MB RGGMONX
29.832 MB RGGMONY

#####################################
#   Total replicat Process Usage    #
#####################################
Number of processes      = 2
AVG Memory usage/process = 29.8359 MB
Total memory usage       = 59.6719  MB



Here’s the example running on Solaris. This configuration has processed over 11,000 transactions per second at peak times, but normally runs around 5000 transactions per second on average.


oracle@sadcsim1db1[CSOTADB1] $ ./gg_memory_usage.sh
OSNAME = SunOS

#####################################
# Individual extract Process Usage #
#####################################
7565.41 MB /ggs/dirprm/e1.prm
5435.1 MB /ggs/dirprm/e2.prm
160.758 MB /ggs/dirprm/p1.prm
156.758 MB /ggs/dirprm/p2.prm
164.758 MB /ggs/dirprm/p3.prm
160.758 MB /ggs/dirprm/p4.prm
160.758 MB /ggs/dirprm/p5.prm
156.758 MB /ggs/dirprm/p6.prm
164.758 MB /ggs/dirprm/p7.prm
160.758 MB /ggs/dirprm/p8.prm
160.758 MB /ggs/dirprm/p9.prm
156.758 MB /ggs/dirprm/p10.prm
164.758 MB /ggs/dirprm/p11.prm
160.758 MB /ggs/dirprm/p12.prm
160.758 MB /ggs/dirprm/p13.prm
156.758 MB /ggs/dirprm/p14.prm
164.758 MB /ggs/dirprm/p15.prm
160.758 MB /ggs/dirprm/p16.prm

#####################################
#   Total extract Process Usage    #
#####################################
Number of processes      = 18
AVG Memory usage/process = 865.147 MB
Total memory usage       = 15572.6  MB

#####################################
# Individual replicat Process Usage #
#####################################
148.688 MB /ggs/dirprm/r1.prm
144.688 MB /ggs/dirprm/r2.prm
144.688 MB /ggs/dirprm/r3.prm
148.688 MB /ggs/dirprm/r4.prm
144.688 MB /ggs/dirprm/r5.prm
144.688 MB /ggs/dirprm/r6.prm
144.688 MB /ggs/dirprm/r7.prm
144.688 MB /ggs/dirprm/r8.prm
144.688 MB /ggs/dirprm/r9.prm
144.688 MB /ggs/dirprm/r10.prm
144.688 MB /ggs/dirprm/r11.prm
144.688 MB /ggs/dirprm/r12.prm
148.688 MB /ggs/dirprm/r13.prm
144.688 MB /ggs/dirprm/r14.prm
144.688 MB /ggs/dirprm/r15.prm
148.688 MB /ggs/dirprm/r16.prm
144.688 MB /ggs/dirprm/r17.prm
144.688 MB /ggs/dirprm/r18.prm
144.688 MB /ggs/dirprm/r19.prm
144.688 MB /ggs/dirprm/r20.prm
144.688 MB /ggs/dirprm/r21.prm
144.688 MB /ggs/dirprm/r22.prm
144.688 MB /ggs/dirprm/r23.prm
144.688 MB /ggs/dirprm/r24.prm
148.688 MB /ggs/dirprm/r25.prm
144.688 MB /ggs/dirprm/r26.prm
144.688 MB /ggs/dirprm/r27.prm
148.68 MB /ggs/dirprm/r28.prm
144.68 MB /ggs/dirprm/r29.prm
144.68 MB /ggs/dirprm/r30.prm
144.68 MB /ggs/dirprm/r31.prm
144.68 MB /ggs/dirprm/r32.prm
144.68 MB /ggs/dirprm/r33.prm
144.68 MB /ggs/dirprm/r34.prm
144.68 MB /ggs/dirprm/r35.prm
144.68 MB /ggs/dirprm/r36.prm

#####################################
#   Total replicat Process Usage    #
#####################################
Number of processes      = 36
AVG Memory usage/process = 145.352 MB
Total memory usage       = 5232.68  MB

Big difference! If you have a busy system, make sure you have memory. You’ll probably have to do testing since the memory requirements will vary depending on both the number of transactions and the size of the transactions.

Here’s the script :


#!/bin/bash

###############################
# determine the OS type
###############################
OSNAME=`uname`

case "$OSNAME" in
  "SunOS")
    echo "OSNAME = $OSNAME"
    ;;
  "Linux")
    echo "OSNAME = $OSNAME"
    ;;
  "*")
    echo "This script has not been verified on $OSNAME"
    exit 1
    ;;
esac

###############################
# set the temp file
###############################
TMPFILE=/tmp/pmem.tmp
if [ -f $TMPFILE ]
then
  rm -f $TMPFILE
fi

################################
# loop over the gg process types
################################
PROCESSES="extract replicat"

for PROCESS in $PROCESSES
do
  FLAG=""
  FLAG=`ps -ef | grep $PROCESS`
  if [ -z "FLAG" ]
  then
    echo "No $PROCESS processes found"
  else
    echo
    echo "#####################################"
    echo "# Individual $PROCESS Process Usage #"
    echo "#####################################"
    case "$OSNAME" in
      "Linux")
        ps -C $PROCESS -O rss > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{print $2/1024, "MB", $12}' | sort -k 2
        ;;
      "SunOS")
        ps -efo vsz,uid,pid,ppid,pcpu,args | grep -v grep | grep $PROCESS > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{print $1/1024, "MB", $8}' | sort -k 2
        ;;
      "*")
        echo "This script has not been verified on $OSNAME"
        exit 1
        ;;
    esac
    rm -f $TMPFILE

    echo
    echo "#####################################"
    echo "#   Total $PROCESS Process Usage    #"
    echo "#####################################"
    case "$OSNAME" in
      "Linux")
        ps -C $PROCESS -O rss > $TMPFILE
        cat $TMPFILE | grep $PROCESS | awk '{count ++; sum=sum+$2; } END \
          { print "Number of processes      =",count; \
          print "AVG Memory usage/process =",sum/1024/count, "MB"; \
          print "Total memory usage       =", sum/1024,  " MB"}'
        ;;
      "SunOS")
        ps -efo vsz,uid,pid,ppid,pcpu,comm | grep -v grep | grep $PROCESS > $TMPFILE
        cat $TMPFILE | awk '{count ++; sum=sum+$1; } END \
          { print "Number of processes      =",count; \
          print "AVG Memory usage/process =",sum/1024/count, "MB"; \
          print "Total memory usage       =", sum/1024,  " MB"}'
        ;;
      "*")
        echo "This script has not been verified on $OSNAME"
        exit 1
        ;;
    esac
    rm -f $TMPFILE
  fi
done

exit 0

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s