Debugging parameter mismatch across RAC database instances with R, dba_hist, and gv$parameter

October 9, 2012

(This article was first published on Bommarito Consulting » r, and kindly contributed to R-bloggers)

Did you find this post useful?  Does your organization need Oracle services?  We can help.

  Much of this morning went into investigating strange ADDM reports on a two-node Oracle RAC database.  For some reason, there were statistically improbable differences between impact percentages that have persisted over a month; in this case, instance 2 would consistently have wildly higher impacts for some optimizer-related activities.  For example:

Finding 3: Hard Parse
Impact is .09 active sessions, 15.46% of total activity.
Hard parsing of SQL statements was consuming significant database time in some

Instances that were significantly affected by this finding:
Number Name Percent Impact ADDM Task Name
------ ------- -------------- --------------
2 RAC2 70.16 Report 20121009D$2
1 RAC1 29.84 Report 20121009D$1

  My first line of research was to review connection strings on the custom applications and  reporting tools that have been put in place.  I assumed that someone had skipped the SCAN address or specified a DSN with SID, not service name.  There must have been some reporting SQL that changed or was poorly auto-generated, causing the optimizer load to differ.  However, this investigation led to no results.

  Next, I tried to trace the issue back through the basics.  The differences were primarily in optimizer activity, so I wanted to look at a time series plot of optimizer_cost in dba_hist_sqlstat by instance.

SELECT dhss.instance_number as instance_number,
    dhss.snap_id as snap_id, 
    AVG(dhss.optimizer_cost) as average_cost, 
    SQRT(VARIANCE(dhss.optimizer_cost)) as stddev_cost 
  FROM dba_hist_sqlstat dhss
  GROUP BY dhss.instance_number, dhss.snap_id
  ORDER BY dhss.snap_id, dhss.instance_number;

  This got some pretty strange results.  I plotted the output in R with ggplot using the lines below.

ggplot(data) +
  geom_point(aes(x=snap_id, y=average_cost, color=factor(instance_number))) + 
  geom_smooth(aes(x=snap_id, y=average_cost, color=factor(instance_number), fill=factor(instance_number)), alpha=0.2) + 
  theme_grey() + 
  scale_x_continuous("Snapshot ID") + 
  scale_y_continuous("Average optimizer cost") + 
  ggtitle("Optimizer cost time series")

Oracle optimizer cost time series by RAC instance

  Instance two has had strictly higher loess-smoothed average optimizer costs, even during snapshots in which the average cost was nearly zero.  It’s almost like they were using entirely different optimizers.  Then it hit me – what if some parameters that affected the optimizer were not synchronized between the databases?  How could I quickly check?

SELECT, p1.value, p2.value FROM gv$parameter p1
  JOIN gv$parameter p2 ON =
  WHERE p1.inst_id = 1
    AND p2.inst_id = 2
    AND p1.value != p2.value
    AND NOT IN ('instance_number', 'instance_name', 'local_listener');

  Lo and behold, optimizer_mode, optimizer_index_cost_adj, optimizer_index_caching, and workarea_size_policy differed between instance 1 and instance 2.  Time to get this fix in to change control review and help get this system back up to full steam.

Did you find this post useful?  Does your organization need Oracle services?  We can help.

To leave a comment for the author, please follow the link and comment on their blog: Bommarito Consulting » r. offers daily e-mail updates about R news and tutorials on topics such as: Data science, Big Data, R jobs, visualization (ggplot2, Boxplots, maps, animation), programming (RStudio, Sweave, LaTeX, SQL, Eclipse, git, hadoop, Web Scraping) statistics (regression, PCA, time series, trading) and more...

If you got this far, why not subscribe for updates from the site? Choose your flavor: e-mail, twitter, RSS, or facebook...

Comments are closed.


Never miss an update!
Subscribe to R-bloggers to receive
e-mails with the latest R posts.
(You will not see this message again.)

Click here to close (This popup will not appear again)