Posts on Tanel Poder Blog

Testing the Datadog Explain Plan Visualizer with Oracle execution plans

Sat, 28 Feb 2026 20:37:33 -0500

Datadog recently released a free online service explain.datadoghq.com where you can paste in your explained SQL execution plans with metrics as JSON text and it visualizes it for you. They support multiple different databases. Hopefully the interactive UI features help you determine where the problem is, without having to mentally parse and scroll through lots of text. This is a topic close to my heart, as you can tell from my “xb” scripts.

New scripts: vstat.sql & dstat.sql for showing Oracle Sysmetric history with 1-minute granularity

Sun, 09 Nov 2025 21:53:07 -0500

Here are two scripts for showing GV$SYSMETRIC_HISTORY contents in a simple tabular way: vstat.sql - show up to one hour of current history at 1-minute granularity from memory awr/dstat.sql - show 1-minute granularity sysmetric history across your entire AWR history The database metrics infrastructure was introduced back in Oracle 10g, so you (and built-in tuning advisors) could use typically needed metrics in a standardized precomputed manner, without having to do various V$ snapshots and performance counter math yourself.

xb.sql and xbi.sql - Explain Oracle Execution Plans Better! (Part 3 - Cardinality Misestimates)

Mon, 03 Nov 2025 01:00:29 -0500

Here’s part 3 of my xb.sql & xbi.sql article series that I started writing back in 2019 :-) Part 1 - Show and format execution plans by querying V$SQL_PLAN data directly Part 2 - Show plan line level “self-values” for A-Time and logical, physical I/Os Part 3 - Show plan line level cardinality misestimate factors (this article) The importance of row-count (cardinality) estimates When I look at complex enough SQLs and optimizer misbehavior, the optimizer cardinality estimates (E-Rows) are the most important optimizer numbers to follow in the execution plan, for understanding the optimizer choices.

tabhisthybrid.sql: Display Oracle hybrid histograms and filter cardinality estimates

Mon, 20 Oct 2025 02:05:56 -0400

I have uploaded an early version of my Oracle hybrid histogram examination script that shows you the estimated row-count (cardinality) numbers when looking up a single value with a simple WHERE col = X condition from a table. You can pull the latest tpt-oracle repo or see it here: https://github.com/tanelpoder/tpt-oracle/blob/master/tabhisthybrid.sql When troubleshooting optimizer’s row-count estimates on tables with histograms, I sometimes end up gradually simplifying the query or pulling out individual tables and WHERE-clauses to see what estimates the CBO comes up with in isolation.

pqr.sql: Generate QR Codes with Pure SQL in PostgreSQL

Sat, 11 Oct 2025 20:31:31 -0400

As my cat woke me up way too early for a Saturday morning, I decided to put all this extra time (and my ChatGPT Pro subscription) into good use and generated a QR-code generator for PostgreSQL, written as a single pure SQL statement. No external libraries or Postgres extension installation needed! The pqr.sql example usage is simple: $ psql -qf pqr.sql -v payload='Hello, World!' There’s also a bit more complex version pqrsafe.

The xCapture and xtop eBPF tools are now in beta, with a demo dataset

Tue, 09 Sep 2025 12:18:32 -0400

Update: I just released xcapture v3.0.3 at P99 CONF, the first proper release of this whole concept! The 20-minute launch video is available for streaming on P99 CONF website. Repo: github.com/tanelpoder/0xtools The title and update say it all, I think it’s time to call the xcapture and xtop toolset ready for use now. I plan to announce the production release (or at least a release candidate) at P99CONF this October.

CatVector Demo Website

Sun, 24 Aug 2025 19:50:26 -0400

I have added a static version of my CatVector demo app for visualizing embedding vectors: tanelpoder.com/catvector/ Previously you had to clone the CatBench repo and run the little Python webserver locally, but now it’s just a publicly accessible website. It loads a few multi-megabyte JSON files into arrays in your browser and then dynamically looks up a specific pet photo that you hover your mouse over in the UI. Depending on your Internet connection, it might take a bit of time for each pet type’s dataset to load.

New Tool: xstack - Completely Passive eBPF Linux Stack Profiling Without Any Tracepoints

Thu, 14 Aug 2025 17:01:35 -0400

I just added another tool into my 0x.tools toolset: With xstack, you can passively sample both kernel stacks and user stacks (as long as framepointers are present) with a pretty minimalistic tool, with no direct impact to your critical application processes! While xstack itself is just a data extraction tool, you probably want to summarize/profile all those thread states and stack profiles. Here’s an example of piping xstack output to flamelens to immediately display a (terminal) flamegraph from the sampled data:

Dimensional Analysis of System Performance with eBPF & xtop: Top for Wall-Clock Time

Tue, 12 Aug 2025 23:46:44 -0400

If you are wondering how the dimensional eBPF data collection works under the hood, check out my earlier post about xcapture: eBPF performance analysis and the 0x.tools repo Summary xtop is like the Linux top tool, but extended with x-ray vision and ability to view your performance data from any chosen angle (that eBPF allows to access). This enables dimensional performance analysis on Linux and tools like top for wall-clock time and much more.

Webinar: Testing High-Performance Vector Search With CatBench on Google AlloyDB Omni

Wed, 02 Jul 2025 04:32:01 -0400

TL;DR: I’m doing a CatBench Vector Search on AlloyDB webinar (hacking session style!) with Google on Wednesday, July 23th at 9am EDT. I was having some chats with Kiran Tailor (of Google) and one thing led to another: We are announcing a webinar/hacking session where I stress-test the latest (Postgres-compatible) Google AlloyDB Omni release using my CatBench (cat benchmarking) vector search stress-test suite! I created the CatBench Vector Search Playground mainly because I wanted to learn some of the “existing business data + AI” opportunities myself.

My Blog Has Turned 18 Years Old

Wed, 18 Jun 2025 14:14:49 -0400

Today, my blog turned 18 years old! I published my first blog entry on June 18, 2007 and have since published over 300 posts. The first blog entry, like most of my others, was about systematic troubleshooting and performance optimization of various (Oracle) database workloads and especially their touchpoint with underlying Unix/Linux operating systems, storage and network I/O. The articles covering interactions between multiple layers of your application stack and how to navigate troubleshooting across all the layers and moving parts have been the most interesting ones for me to write.

Catbench Vector Search App has Postgres Query Throughput and Latency Monitoring Now

Fri, 30 May 2025 14:18:01 -0400

CatBench is a Postgres + PgVector demo application that uses 25k pet photos from a Kaggle dataset for demonstrating how the similarity search features work together with the rest of your application code & schema in the backend (HammerDB Order Entry). [Jump to the new stress test + monitoring section] CatBench Interactive UI You can navigate around the recommendation engine for Cat Purchases UI, click on cat photos and find product recommendations, based on what other similar-looking cats have purchased.

Using Postgres pg_test_fsync tool for testing low latency writes

Tue, 27 May 2025 20:31:34 -0400

Here’s a useful tool for quickly testing whether a disk (or a cloud block store volume) is a good candidate for your database WAL/redo logs and any other files that require low latency writes. The pg_test_fsync tool is bundled with standard Postgres packages, so no extra installation is needed. You don’t actually have to use Postgres as your database, this tool’s output is universally valuable for any workload requiring fast writes.

Find which Oracle SQL_ID hits the SQL*Net break/reset to client event

Wed, 14 May 2025 10:00:08 -0400

Oracle’s SQL*Net break/reset to client wait event shows up whenever Oracle (SQL, PL/SQL call, etc) returns an ORA- error back to the client instead of the normal expected results (and ORA-0 under the hood): SQL> @oerr 0 ORA-0000: normal, successful completion I explained this behavior a bit more in my 2008 post about Oracle’s SQL*Net break/reset to client post. Despite the wait event name, this break/reset wait doesn’t mean that you have a network problem, but more likely some sort of an application problem (like trying to insert duplicate values into a unique column, etc).

New Tool: lsds - List All Linux Block Devices and Settings in One Place

Fri, 09 May 2025 10:23:21 -0400

When dealing with disks and I/O things on Linux, you’d regularly run commands like lsblk, lsscsi, nvme list, etc. All of them tend to report a different set of information, so I ended up running multiple commands and correlating their output based on the device name or number. And then I had to run commands like these, to get extra info about the current OS-level configuration settings for specific disks:

Optimizing eBPF I/O latency accounting when running 37M IOPS on a 384-CPU server

Tue, 29 Apr 2025 21:42:03 -0400

In this post I will introduce a much more efficient method for accounting block I/O latencies with eBPF on Linux. In my stress test, the “new biolatency” accounting method had 59x lower CPU and probe latency overhead compared to the current biolatency approach. So I couldn’t help it and ended up putting 21 NVMe SSDs into one of my homelab servers. 8 of them are PCIe5 and the remaining 13 are PCIe4.

xCapture v3: Linux Performance Analysis with Modern eBPF and DuckDB

Wed, 23 Apr 2025 16:04:06 -0400

To capture your attention, this is what you get: Table of contents What is xCapture? xCapture with modern eBPF Modular and decoupled at multiple levels Dimensional performance data analysis with DuckDB Top wall-clock time used by all threads Regular expressions in the filter Syscalls against network socket file descriptors What is the performance overhead? Performance overhead of active tracking probes Userspace processing overhead Running xcapture What’s next? What is xCapture? 0x.

Core Function Words Translated between English, Spanish, German, French and Swedish

Fri, 18 Apr 2025 17:38:34 -0400

What is this? When learning a new (human) language like Spanish or French, I’ve found that not knowing the various common “connecting words” of a sentence, like “from, to, it, at, with, without, under, on, less, more, is, not” and so on is a bottleneck for me. I might know the words like “dog” “cat” “food”, but when just starting my learning, not remembering the common “connecting words” always adds a recalling “tax” or bottleneck for me when trying to speak.

When eBPF task->stack->pt_regs reads return garbage on the latest Linux kernels, blame Fred!

Fri, 28 Feb 2025 19:57:27 -0500

TLDR; Starting from Linux kernel version 6.9 on x86_64, there’s a new config option CONFIG_X86_FRED enabled and it adds 16 bytes to the starting point of a task’s kernel stack area, so you’ll need to account for this extra padding in your “raw” kernel stack & pt_regs lookup code. Update: It turns out that there’s a bpf_task_pt_regs() helper function available in newer kernel/eBPF versions and it does the same offset math for you, if you can retrieve your task using the bpf_get_current_task_btf() BTF-enhanced helper that is data structure aware.

TracepointArgs: List Any Linux Tracepoint with Their Arguments, Datatypes and Related Structs

Thu, 30 Jan 2025 08:13:29 -0500

This should be something useful for Linux kernel explorers and eBPF nerds! Last year I released a tool called syscallargs that walked through the Linux /sys/kernel/debug/tracing/events/syscalls directory tree and allowed you to query and list available system calls in your current system from the command line. And it printed out syscall argument names and their datatypes, so it has helped me avoid opening man-pages (or web-pages) a few times. Later I realized that since many other kinds of kernel events and tracepoints are also presented in the same /sys directory, in the same format as syscall events, we can examine all of them using the same approach.

CatBench Vector Search Playground

Sun, 26 Jan 2025 15:49:58 -0500

Last year, I published a simple demo app that visualizes embedding vectors of a bunch of cat, dog and airplane photos as heatmaps. It’s a simple concept and does not have much practical use. I built it mostly for fun, but also for some “learning by doing” too. Today I’m releasing a completely new version of my CatBench Vector Search Playground app! https://github.com/tanelpoder/catbench With this new app, you can interactively navigate through multiple Postgres-based vector similarity search examples and immediately see which SQL statements were executed in the backend and their execution plans with performance metrics.

Embedding Vectors of the Same Image Rotated Over 360 Degrees

Sun, 12 Jan 2025 22:34:14 -0500

Last year I wrote an article about visualizing embedding vectors of a variety of different pictures as heatmaps. I used TorchVision with ImageNet1K_V1 model for encoding a bunch of different cat, dog and plane images into their embeddings, 1000 floating-point values in each resulting vector. I used the generate_embeddings.py script for that. This whole exercise is just for my own learning (and fun), so it’s ok if there’s no other practical value coming from this work.

SQL Plan Execution FlameGraphs with Loop and Row Counts

Fri, 03 Jan 2025 23:12:10 -0500

Back in 2018 I wrote about how to apply Brendan Gregg’s FlameGraphs for visualizing SQL plan execution time 1, using the plan node-level context and timing data that some database engines provide. This approach is not designed to replace the traditional plan display methods that are packed with lots of additional metrics and details. Nevertheless, in case of large execution plans, SQL plan flamegraphs give you a quick overview of both the plan structure, tables and operations involved and where (in which plan subtrees) most of the plan execution time is spent.

Embedding Vectors vs. Vector Embeddings

Sat, 07 Dec 2024 20:38:11 -0500

Disclaimer: I’m not an ML expert and not even a serious ML specialist (yet?), so feel free to let me know if I’m wrong! It seems to me that we have hit a bit of an “on-premises” vs. “on-premise” situation in the ML/AI and vector search terminology space. The majority of product announcements, blog articles and even some papers I’ve read use the term vector embeddings to describe embeddings, but embeddings already are vectors themselves!

Video: Testing the Silk Platform in 2024 Interview (12 minutes)

Thu, 28 Nov 2024 14:52:59 -0500

Here’s a 12-minute video of our chat with Silk VP of Product Tom O’Neill about my recent testing of the Silk Platform in Google Cloud. In this interview we cover some high level points, conclusions and talk a little bit about the future. If you want to read all the technical details and some interesting references, see the links below: Testing the Silk Platform in 2024: Achieving 20 GiB/s I/O Throughput in a Single Cloud VM (2024) Testing The Silk Platform - Hands-On Technical Analysis of High-Performance I/O in the Cloud (2021) NVIDIA 800Gbit network cards already on display (2024) 1.

Testing the Silk Platform in 2024: Achieving 20 GiB/s I/O Throughput in a Single Cloud VM

Tue, 05 Nov 2024 19:32:23 -0500

Hands-on technical analysis of a novel data platform for high-performance block I/O in the cloud, tested by Tanel Poder, a database consultant and a long-time computer performance nerd. Index Background and motivation Scalable Architecture by Design Enterprise Features Testing Results I/O Latency 1.3 Million IOPS Lessons Learned Summary Background and Motivation Back in 2021, my old friend Chris Buckel (@flashdba) asked me if I would like to test out the Silk Data Platform in the cloud and see how far I could push it.

Visualizing and Comparing Embedding Vectors as Heatmaps (Videos)

Mon, 23 Sep 2024 09:32:27 -0400

Following my earlier Visualizing and Comparing Embedding Vectors as Heatmaps article, here are the launch & demo videos. You might want to read this article first, these short demo videos should make more sense then! Visualizing Vision Transformer Embedding Vectors as HeatMaps In this video I’ll peek around cat vectors and then find outliers from both the cat photo dataset and aircraft dataset. Installing CatBench Vector Visualizer App and Technical Details The CatBench GitHub repo is here:

Visualizing and Comparing Embedding Vectors as Heatmaps

Fri, 20 Sep 2024 21:16:34 -0400

An interactive demo video is here: Part 2. If you already know what embeddings are, you can jump right to: Vector Heatmap Visualization Background I am currently testing various RDBMS vector search options and their usability & performance. Next week I will publish an early version of my CatBench toolset that will eventually evolve into a full end-to-end test (and fun!) suite for high performance AI/ML pipelines, with things like streaming image recognition using Vision Transformers on GPUs, followed by vector searches in target databases and more.

Tracking, not Tracing, Linux Thread Activity for Complete System Visibility (eBPF Summit)

Thu, 12 Sep 2024 20:55:07 -0400

Here’s my talk from eBPF Summit 2024. Video (10 minutes) The 0x.tools Extended Task State Sampling approach provides a new observability signal - wall-clock time of all active threads’ activity in your system. You can see both total workload demand of your apps and also drill down deep into individual threads’ activity when needed. This method and toolset give you a single data source with many directly linked fields of information about what your application & OS threads are doing and why.

Python Supports Profiling with Perf - How Does It Work?

Wed, 31 Jul 2024 01:14:56 -0400

Starting from Python 3.12, you can instruct Python to enable direct perf profiling support for your Python code. This feature currently works only on Linux, as it communicates the relationships between Python-level interpreted function calls and machine-code level return pointers (for stack unwinding) via perf map files. The Linux perf tool knows how to read these files and maps the return pointers in a thread’s stack to the corresponding Python-level symbols (similar to what JVM JIT code profiling with Perf maps does).

Can a User Process Consume Kernel Mode CPU Without a Syscall?

Wed, 10 Jul 2024 09:33:13 -0400

Yes! Various traps like page faults may cause your user process to be switched into kernel codepath even when the process is minding its own business in the userspace. It just needs to do something that causes such a trap, like touch a new page of memory in its virtual memory address space, that hasn’t been “fully materialized” yet in the kernel memory structures. For example, after starting up an Oracle database instance that uses a large amount of shared memory, there’s one process that burns CPU for a while:

Next Big Thing: X-Ray Vision for Linux Systems with eBPF

Fri, 21 Jun 2024 20:14:02 -0400

I recently published what I have been working on for the past 5 years. No, this is not about AI or a yet another startup - I’m still a big fan of NI (natural intelligence) and what I reveal is open source and free for everyone to use. I’m not gonna sell you any products, only ideas! This is a new approach and a supporting toolset for understanding system behavior and its performance across all your applications and data center & cloud footprint.

Syscallargs: List All Linux System Calls With Their Arguments

Thu, 13 Jun 2024 23:50:24 -0400

This blog post is not about tracing program system calls, but about programmatically extracting all system calls and their accepted arguments in your current system. If you want to read more about tracing system calls, here’s my tutorial using strace for troubleshooting a real life scenario: Troubleshooting Linux SSH Login Delay - Why does logging in always take 10 seconds?1 Update: I have also created a similar program tracepointargs for listing any Linux tracepoint with their arguments, datatypes and related structs.

RTX4090 converted into an eGPU

Fri, 07 Jun 2024 06:59:31 -0400

One evening back in January I finally had enough of thermal issues within my homelab server. You know, every time the computer fans make more noise than I think they should, I can’t help but investigate! Also, the RTX4090 is so thick that it takes 3 PCIe slots worth of space on a typical motherboard - and you also need space for the airflow! I was using the same machine that achieved 11M IOPS & 66 GB/s IO on a Single ThreadRipper Workstation back in 2020.

Oracle Hint Scopes and Join Level Cardinality Hint

Thu, 06 Jun 2024 22:16:36 -0400

Thanks to Tomasz Sroka’s pull request for my hint.sql in the TPT repo and blog entries by Sayan 1 and Jonathan 2, we now have a HINT_SCOPE field in my hint.sql script too! SQL> @hint merge NAME VERSION VERSION_OUTLINE INVERSE HINT_SCOPE ----------------------- ----------- --------------- -------------- -------------- MERGE_CONST_ON 8.0.0 STATEMENT MERGE_AJ 8.1.0 8.1.7 QBLOCK MERGE_SJ 8.1.0 8.1.7 QBLOCK MV_MERGE 9.0.0 QBLOCK MERGE 8.1.0 10.1.0 NO_MERGE QBLOCK OBJECT NO_MERGE 8.0.0 10.1.0 MERGE QBLOCK OBJECT USE_MERGE_CARTESIAN 11.

New Commenting System

Wed, 05 Jun 2024 16:49:45 -0400

I switched my blog’s commenting system to use Giscus. It’s still backed by Github (Discussions) as “storage”, but it doesn’t request wide Github permissions from commenters (like Utterances used to do). I didn’t migrate old comments over, but they’re still available here: https://github.com/tanelpoder/blog-comments/issues/ For current/latest comments, just scroll to the bottom of the blog entries (or browse the corresponding GitHub Discussions page): https://github.com/tanelpoder/blog-comments/discussions/

Oracle Shared Pool Internals: Allocated Chunk Status Indicators in Heap Dumps

Fri, 08 Mar 2024 12:56:46 -0500

Over time, Oracle has been adding more contextual information into each allocated heap memory chunk, to make it easier to see what for your heap memory is used. This instrumentation is used for private (PGA,UGA,etc) heaps too, but this article focuses only on shared pool heaps. A few examples from past are: Library cache object hash values added to allocated shared pool chunk names (2010) The above technique is used for some other allocation types too, like KKSSP^NNN allocations that are used for “session pages” that hold various tiny memory structures like library cache lock (and pin in older DB versions) that a session must allocate when accessing library cache objects.

Oracle Index Parallel Degree Fix After Parallel Creation or Rebuild

Mon, 12 Feb 2024 23:55:14 -0400

Back in 2007, my third blog entry was about a parallel index building gotcha, a warning about building & rebuilding indexes in parallel. After completing the build, Oracle left the parallel degree in place for that index in data dictionary. Thus, after building a great new index over some weekend, many of your OLTP queries could have picked a parallel plan afterwards! Whenever creating/rebuilding any indexes, I had a habit of writing up both the CREATE INDEX .

Report Current Oracle Session Wait Chains from GV$SESSION without ASH

Sat, 24 Sep 2022 18:08:14 -0400

You probably are already familiar with my ash_wait_chains scripts that use CONNECT BY for walking through the blocking_session fields in ASH and report “wait chains”, signatures of complex waits involving other database sessions. Since ASH is “just” a history of V$SESSION, it’s sensible to assume that one could sample the current wait chains just by querying the GV$SESSION views, instead of summarizing history from ASH. And indeed it’s possible - here’s my Session-Wait-Chains script swc.

Advanced Oracle Troubleshooting Guide, Part 13: OStackProf for Linux, Unix & MacOSX Clients

Sat, 17 Sep 2022 00:41:48 -0400

About 14 years ago I published the OstackProf script that ran on Windows sqlplus clients (the Oracle server could run on any platform that supported ORADEBUG SHORT_STACK). I don’t use Windows almost at all nowadays, so it’s about time to publish the same functionality for MacOSX, Linux, Unix sqlplus clients. This is mostly useful when hacking dev systems, troubleshooting test systems and in rare cases, carefully troubleshooting non-fatal processes in production too.

Testing Oracle's Use of Optane Persistent Memory, Part 2 - Fast Log File Sync Waits

Fri, 15 Jul 2022 01:10:23 -0400

Update: Just as I started blogging about the Optane technology, it turns out that Intel is shutting down the Optane business (after bringing the latest-gen “Crow Pass” Optane memory to market - probably just for satisfying existing commitments). While not exactly the same thing, keep an eye on the new CXL standard developments to see where the data center/cloud infrastructure world is going (CXL is basically cache-coherent disaggregated memory access protocol via PCIe5.

Testing Oracle's Use of Optane Persistent Memory, Part 1 - Low Latency Commits

Thu, 28 Apr 2022 00:54:38 -0400

MySQL Now Shows its Thread Names at OS Level for Better Troubleshooting

Thu, 17 Mar 2022 14:19:40 -0400

Starting from v8.0.27, the MySQL daemon shows its thread names at the OS level too, for better troubleshooting. The MySQL 8.0.27 release notes say this: To assist monitoring and troubleshooting, the Performance Schema instrumentation is now used to export names of instrumented threads to the operating system. This enables utilities that display thread names, such as debuggers and the Unix ps command, to display distinct mysqld thread names rather than “mysqld”.

xb.sql and xbi.sql - Explain Oracle Execution Plans Better! (Part 2)

Thu, 17 Mar 2022 01:25:20 -0400

In the previous post in this series, I showed how my scripts xb.sql and xbi.sql (eXplain Better and eXplain Better, by sqlId) help you see Oracle execution plan details in a more complete way. Here are the features that I currently plan to cover: More detailed access/filter predicate info next to plan tree data (Part 1) More intelligent row-source level execution statistics presentation (Part 2, this post) Calculate Optimizer Cost misestimation factor and demo its effects (Part 3, coming soon) More intelligent row-source level execution statistics presentation Look into the following example output by DBMS_XPLAN.

My New Learning Platform and What I'm Up To in 2022

Mon, 14 Mar 2022 22:12:31 -0400

Tanel at Mount Denali in Alaska (2021) In the beginning of every year I travel somewhere and take some time to think about the future and what I should be working on next. The 2020/2021 years were full of all kinds of distractions for me (and many of you). I got somewhat sick a couple of times too (although never tested positive for COVID) and the full recovery to 100% took time.

Achieving 10 GB/s Scanning Rate in a Single Cloud VM

Thu, 18 Nov 2021 13:52:00 -0400

You may remember my previous article about high-performance I/O for cloud databases, where I tested out the Silk Platform and got a single Azure VM to do over 5 GB/s (gigaBytes/s) of I/O for my large database query workload. The magic was in using iSCSI over the switched cloud compute network to multiple “data VMs” at the virtualized storage layer for high I/O throughput and scalability. Well, the Silk folks are at it again and Chris Buckel (@flashdba) just sent me screenshots of similar test runs using the latest Azure “v5” instance as the single large database node and they achieved over 10 GB/s scanning rate for reads and 6 GB/s write rate with large I/Os!

Testing The Silk Platform - Hands-On Technical Analysis of High-Performance I/O in the Cloud

Mon, 18 Oct 2021 06:30:00 -0400

Update: 3 years after writing this article, I tested the latest Silk Platform in 2024, on Google Cloud Platform, with impressive results. In my tests, I got to over 20 GiB/s with large reads, 1.3M IOPS with small 8kB random reads and 10 GiB/s sustained write rate! Both articles are worth a read. Hands-on technical analysis of a novel data platform for high-performance block I/O in the cloud, tested by Tanel Poder, a database consultant and a long-time computer performance geek.

Virtual Conference: Troubleshooting Very Complex Oracle Performance Problems

Fri, 19 Mar 2021 15:22:07 -0400

Learn to troubleshoot some of the most complex Oracle performance problems - real life case studies by Tanel Põder I already announced my latest virtual conference in social media, but will write a short blog entry here too (people who are following my blog via mailing list or RSS will get the update too). This year’s topic is: Troubleshooting Very Complex Oracle Performance Problems. I will deliver 4 case studies (based on real life events) where I will dive very deep, even deeper than in my previous years’ Advanced Oracle Troubleshooting training class!

Do Not Use '>' in Your Command Prompt (and How to Stay Safe in Shell)

Tue, 16 Mar 2021 16:30:29 -0500

Over the years of troubleshooting performance problems in the Unix/Linux world, I have seen multiple cases where a regularly used command line tool in a customer server just stops working for some reason. The tool just returns immediately, doing absolutely nothing. No output printed, no coredumps and the exit code is zero (success!). This article walks you through a couple of such incidents and in the end I explain how I avoid accidentally doing bad stuff in production in general.

Is Your Linux Version Hiding Interrupt CPU Usage From You?

Mon, 15 Feb 2021 00:00:35 -0500

TL;DR: Some Linux distros and even different kernel versions within a distro have disabled IRQ time accounting. In such case, your monitoring tools will report zero time spent in hardware interrupt handlers (shown as %hi, %irq, hiq, etc). It’s easy to check how your kernel is behaving by looking at /proc/stat and you can still measure interrupt CPU usage using perf and a little creativity. Index Index Problem Explanation Kernel configuration options Measuring IRQ time with perf Different distros and kernel versions Summary Here’s a reminder that your Linux kernel may be not breaking down hardware interrupt CPU usage separately in its /proc/stat CPU time reporting, depending on the kernel’s compile-time settings.

Video: High Performance Block I/O on Linux

Sun, 07 Feb 2021 15:58:57 -0500

Here’s the High Performance Block IO on Linux hacking session video. It’s based on my recent article about achieving 11M IOPS & 66 GB/s IO on a single ThreadRipper workstation. Enjoy! Towards the end of this video, I also explain how I got up to 11.5M IOPS without having to keep CPUs 100% busy, with batched I/O submission and completion checks I reduced the CPU usage pretty noticeably. Also, with properly configuring I/O polling queues in the Linux kernel, I got close to 10M IOPS by using just 10 CPUs (leaving almost 70% of CPU time idle).

Achieving 11M IOPS & 66 GiB/s IO on a Single ThreadRipper Workstation

Fri, 29 Jan 2021 07:40:33 -0500

TL;DR Modern disks are so fast that system performance bottleneck shifts to RAM access and CPU. With up to 64 cores, PCIe 4.0 and 8 memory channels, even a single-socket AMD ThreadRipper Pro workstation makes a hell of a powerful machine - if you do it right! Update 1: Since 2021, networks have gotten faster and now it is possible to have high-speed reliable remote I/O even in cloud VMs!

Oracle Wait Event Name vs. Display Name

Tue, 12 Jan 2021 22:03:08 -0500

Here’s a little SQL query that lists all Oracle wait events where Oracle kernel developers have added additional “human readable” descriptions. Since the original wait event names shouldn’t be changed for compatibility with existing tools, then a new column DISPLAY_NAME was added: SQL> SELECT name, display_name FROM v$event_name WHERE name != display_name; NAME DISPLAY_NAME ----------------------------------------- ---------------------------------------------------------------- control file sequential read control file read control file single write control file write log file sequential read log file multiblock read log file single write log file header write log file parallel write log file redo write log buffer space log buffer full - LGWR bottleneck log file sync commit: log file sync LGWR real time apply sync standby apply advance notification db file sequential read db single block read db file scattered read db multiblock read db file single write db single block write db file parallel write db list of blocks write db file parallel read db list of blocks read wait for possible quiesce finish quiesce database completion concurrent I/O completion online move datafile IO completion datafile copy range completion online move datafile copy range completion Image redo gen delay redo resource management Long operation CF pause CF txn pause DFS db file lock quiesce for datafile offline datafile move cleanup during resize online move datafile resize cleanup From the above highlighted examples you can see that the new “display name” column uses less confusing names for some wait events.

Measuring Java JVM thread activity in Linux with task state sampling

Sat, 28 Nov 2020 17:52:09 -0500

FYI: I will be speaking about similar topics at P99Conf in Oct 2023 (online, free event) There are plenty of JVM profilers out there, but before attaching with a profiler, you can get a high-level overview by just sampling the Linux /proc/PID/comm and /proc/PID/stat fields to see which JVMs threads are most active and whether they’re burning CPU or are blocked by some I/O. When creating threads in Java, you can name them using the following syntax: Thread t = new Thread("MyThread") or thread.

Reasons why SELECT * is bad for SQL performance

Tue, 24 Nov 2020 22:14:58 -0500

Here’s a list of reasons why SELECT * is bad for SQL performance, assuming that your application doesn’t actually need all the columns. When I write production code, I explicitly specify the columns of interest in the select-list (projection), not only for performance reasons, but also for application reliability reasons. For example, will your application’s data processing code suddenly break when a new column has been added or the column order has changed in a table?

High System Load with Low CPU Utilization on Linux?

Fri, 20 Nov 2020 11:31:58 -0500

In this post I will show you how to break down Linux system load by the load contributor or reason. You can drill down into the “linux system load in thousands” and “high system load, but low CPU utilization” problem patterns too. Introduction - terminology Troubleshooting high system load on Linux Drilling down deeper - WCHAN Drilling down deeper - kernel stack How to troubleshoot past problems Summary Further reading Introduction - Terminology The system load metric aims to represent the system “resource demand” as just a single number.

Hacking Session: 7 Levels of Hint Invalidity in Oracle SQL

Thu, 21 May 2020 12:00:00 -0400

As I promised in the end of the last hacking session (about Oracle SQL Monitoring, I will run another one on 2. June 2020. This one will be a deep dive into Oracle hint usage and various scenarios of hint (in)validity. I have too much material in my full-week Advanced Oracle SQL Tuning training, so I’m moving some “narrow deep dives” out and make available for free, so that we could spend more time actually tuning SQL during the class.

Hacking Session: Oracle SQL Monitoring - Understanding the Execution Timeline column and CPU usage

Fri, 08 May 2020 17:00:00 -0400

Update: I have uploaded the videos of this hacking session to YouTube. I will run another hacking session that has been in my mind for a while. It will be a pretty narrow deep-dive into one of the SQL Monitoring report’s columns that is not well explained: Oracle SQL Monitoring - Understanding the Execution Timeline column In this hacking session, I will explain how to read the “Execution Timeline” column in the SQL Monitoring reports.

Virtual Conference: Systematic Oracle SQL Optimization Interview Videos

Wed, 08 Apr 2020 15:58:48 -0400

Update: I have added an interview with Jonathan Lewis to YouTube too. We talk a bit about the last 10 years and Jonathan describes what he’ll be talking about in his virtual conference talk: I will add a few more videos in the coming days. The conference interview playlist URL is here. You can sign up for the conference here. See you soon!

Oracle Fatal Background Processes

Tue, 24 Mar 2020 03:11:16 -0400

When talking about Oracle background processes, there’s a term/qualifier “fatal” background process. This means that when one of these background processes crashes, then whoever detects the process disappearance (PMON or LGWR or CLMN possibly), will shut down the instance as it cannot function normally anymore. Not all background process crashes take down the whole instance, for example processes like J000 and P000 are technically background processes (daemons disconnected from network), but their crash won’t take down the instance.

Virtual Conference Update: Systematic Oracle SQL Optimization in 2020

Mon, 23 Mar 2020 17:27:40 -0400

The current state of affairs in the entire world have radically changed since last month when we announced the next Oracle SQL Optimization event in the virtual conference series. We’ve been keeping an eye on the trajectory of the developments and decided to make some changes too. So, here’s the update: we are going to run the Oracle SQL Optimization in 2020 virtual conference twice! Additional dates These are the existing + new dates for the event:

Oracle Shared Pool Internals: List Chunk Position in the LRU List

Wed, 04 Mar 2020 01:11:29 -0500

There was a discussion at Oracle-L about shared pool chunk eviction and I thought I’d show how to see a shared pool memory chunk’s position in the LRU list yourself. This is a post mostly about Oracle internals and less about practical every day troubleshooting. There’s a view X$KGHLU (@kghlu.sql script) that shows you a summary of shared pool (sub)heap allocations and how many of these chunks are in the “transient list” (used once) vs.

SchedLat: a Low Tech Script for Measuring Process CPU Scheduling Latency on Linux

Wed, 26 Feb 2020 23:18:52 -0500

As you may know, I like to use low tech scripts when possible, ideally such ones that don’t even require you to be root. I prefer simple and “boring” tools as a starting point simply because of the production reality at my customers in traditional enterprise IT. Systems where RHEL6 (and clones) seem to be the most common distros (with Linux kernel 2.6.32) and it’s not too unusual to see RHEL5 occasionally either.

Announcing My Retirement (in 2030)!

Wed, 19 Feb 2020 02:33:00 -0500

I hereby announce that I will retire in 2030 and replace myself with a computer! I am serious. This is probably the clearest mission statement I’ve ever come up with in my career! This is essentially my statement of direction for the next 10 years. There will be a series of posts about what I am up to and where I’m going. My plan for the next 10 years When the year 2020 arrived, I spent a month thinking about the future and what I want to be working on for the next decade.

ORA-12850: Could not allocate slaves on all specified instances error and a workaround for my scripts

Wed, 12 Feb 2020 00:04:11 -0500

My ASH wait chains scripts started returning the ORA-12850 error on Oracle 19c RAC installations: SQL> @ash/dash_wait_chains program2||event2 1=1 sysdate-1 sysdate -- Display ASH Wait Chain Signatures script v0.5 BETA by Tanel Poder ( https://tanelpoder.com ) , REPLACE(SYS_CONNECT_BY_PATH(program2||event2, '->'), '->', ' -> ')||CASE WHEN CONNECT_BY_ISLEAF = 1 AND d.blocking_session IS NOT NULL THEN ... ERROR at line 63: ORA-12850: Could not allocate slaves on all specified instances: 2 needed, 1 allocated While this error message sounds like a resource shortage issue, it is actually often due to the remote instance(s) not being able to reproduce the exact same execution plan as in the query coordinator instance.

Oracle Performance Troubleshooting Without OS Access, Part 1: Identifying CPU Scheduling Latency

Mon, 10 Feb 2020 22:27:51 -0500

As running Oracle databases as (partially) managed services in the cloud has become quite popular, I thought to start a little series about troubleshooting Oracle (performance) stuff when not having OS access. This will help with such cloud services, but also in cases where your team just doesn’t have convenient OS access due to some separation of privileges reasons. The first example is my schedlat.sql script that uses X$KSO_SCHED_DELAY_HISTORY to list sampled process scheduling latency details with a 15 minute history.

Video: Troubleshooting Complex Oracle Performance Problems (Jan 2020)

Sun, 02 Feb 2020 22:52:00 -0500

I have uploaded the 3-hour hacking session video into my YouTube channel in 2 parts. Introduction and using ASH + wait chains for troubleshooting performance spikes A case study of a complex Oracle performance problem The slides are in SpeakerDeck: https://speakerdeck.com/tanelpoder/troubleshooting-complex-oracle-performance-problems FYI there were a few questions about using such techniques without Oracle’s built-in ASH (due to licensing and Oracle Standard Edition). You can look into the Simulated Oracle ASH (OraSASH) package and there’s also my ashtop.

Hacking Session: Troubleshooting Complex Oracle Performance Problems (January 2020)

Thu, 30 Jan 2020 00:00:00 -0400

As promised, I will run one more hacking session in the end of January. I aim to make this one more practical than some of the past internals geek-dives :-) The topic: Troubleshooting Complex Oracle Performance Problems with Tanel Poder Date & Time: Thursday 30 January 2020 (today!) 12pm-1:30pm ET Registration: https://attendee.gotowebinar.com/register/7847585427420817931 Description: In this session I will first present some slides about a complex performance issue where the initial symptoms point somewhere else than the root cause.

help.sql: Show TPT Script Purpose and Syntax

Wed, 15 Jan 2020 10:07:33 -0700

If you use my TPT scripts for Oracle troubleshooting, I have some good news for you. I had created a help.sql script years ago for showing me a quick index of my scripts, with syntax examples, straight from sqlplus command line. BUT, it was just an empty template, I never got to populating help.sql to cover my most used scripts. Recently Tomasz Sroka, who had attended my Oracle troubleshooting training, mentioned that he had taken matters into his own hands and documented about 100 of my scripts in the help command.

Oracle State Objects and Reading System State Dumps Hacking Session Video

Thu, 09 Jan 2020 00:47:23 -0500

I have uploaded the 3 hour hacking session video about Oracle instance state objects and how to read Oracle systemstate dumps to my YouTube channel: Note that as I just uploaded the video, you may want to wait for a few hours until YouTube publishes HD version (currently it’s available only at 360p, but that should change soon). The slides are in Slideshare: https://www.slideshare.net/tanelp/oracle-state-objects-and-system-state-dumps/ Enjoy! :-) Also, big thanks to Kishy Kumar (who’s deeply involved in building this advanced database engine functionality at Oracle) for reviewing my work and giving feedback!

New scripts: xb.sql and xbi.sql - Explain Oracle Execution Plans Better! (Part 1)

Fri, 04 Oct 2019 11:25:20 -0400

Here’s a blog entry about some “new” Oracle execution plan displaying scripts that I’ve had since 2011 or so - I just tidied them up recently and added some improvements too. My aim in this blog post is not to go deep into SQL tuning topics, but just show what these scripts can do. The new scripts xb.sql and xbi.sql (eXplain Better and eXplain Better, by sqlId) are in my GitHub repo.

Dump Current Oracle SQL Execution Plan Line ID Using Oradebug

Mon, 08 Apr 2019 11:03:43 -0400

This post is written mostly for fun, not for practical everyday usability (enjoy!). Introduction There was a question on Twitter about alternative options for retrieving the SQL execution plan line ID that Oracle is currently executing. Normally Oracle’s Active Session History shows this info in its PLAN_LINE* columns as I have explained in my asqlmon.sql post. does anyone knows if I can select the equivalent from ASH.SQL_PLAN_LINE_ID for current session? (something like v$session or x$.

pSnapper: Linux Process Snapper Page and Videos

Wed, 03 Apr 2019 16:03:59 -0400

I have created a new page & uploaded videos of my Linux Process Snapper hacking session: Resources Homepage: https://tp.dev/psnapper GitHub: https://github.com/tanelpoder/psnapper Videos Video 1: Exploring /proc Filesystem & System Calls Hacking Session Video 2: Linux Process Snapper Demo Slide(s) The only slide I showed during the hacking session is this: As I didn’t get to cover some planned Linux tracing topics today, I guess this means we’ll have another hacking session some day!

Announcing Linux Process Snapper tool and a Linux Troubleshooting Hacking Session (3rd April 9am PDT)

Sun, 31 Mar 2019 23:53:28 -0400

As I’m having fun preparing for my Linux troubleshooting training, I’ll also do another informal hacking session to give you an idea of what’s coming up! This week’s hacking session is deliberately without any slides, planning or much structure, but with lots of fun and … hacking! This session will be about some Linux tools I use for my everyday work and I’ll even publish an early version of a new open source Linux performance tool!

What Caused This Wait Event: Using Oracle's wait_event[] tracing

Fri, 29 Mar 2019 11:30:04 -0400

I talked about a new diagnostic event wait_event[] back at OakTableWorld 2014. Oracle introduced this feature in version 12.1. But since there’s only a camera recorded video of that talk, I’ll document some examples here. Attaching Actions to Wait Events The new wait_event[] event name allows you to attach actions, like tracing, to wait events. The action runs whenever a wait event ends. It behaves like Oracle’s SQL Trace that writes out trace lines for wait events only when the wait ends.

Tutorial: Troubleshooting Linux SSH Login Delay - Why does logging in always take 10 seconds?

Thu, 07 Mar 2019 12:16:58 -0500

As I’m delivering my Linux Troubleshooting training soon, I am going to blog about some typical issues and techniques we’ll troubleshoot in the class too. I’ll start from a relatively simple problem - logging in to a server via SSH always takes 10 seconds. The delay seems to be pretty constant, there don’t seem to be major network problems and the server is not overloaded. Yet, remote logins always take 10 seconds.

Practical Linux Performance & Application Troubleshooting Training Details and Table of Contents

Mon, 04 Mar 2019 15:16:07 -0500

As I announced last year, I’m running a new online Practical Linux Performance and Application Troubleshooting training class, starting on 29. April 2019. The emphasis of this class is on practical application troubleshooting and Linux OS performance tuning techniques. We’ll be using scripts and tools that you can use on any Linux machine whenever a problem happens, without having to first install a complex layer of monitoring tools. At the same time, we will go way beyond what the classic sysadmin tools (like vmstat, iostat, top) have to offer.

Scripts for Drilling Down Into Unknown Optimizer Changes

Thu, 08 Nov 2018 22:47:32 -0500

When I delivered my Advanced Oracle SQL Tuning training last week, I jumped into a free-form demo to show that the PARALLEL hints don’t really force anything other than adjusting the cost of parallelizable operations for CBO evaluation as usual. If a serial plan (or branch) still has a lower cost than the respective parallel processing cost, then serial execution would be used there. But when demoing this, I got a surprise.

Visualizing SQL Plan Execution Time With FlameGraphs

Sun, 28 Oct 2018 01:26:02 -0400

Update: I wrote a follow-up article about adding Loop Counters and Row Counts to SQL Plan FlameGraphs. I also show some old examples of “Bookshelf Graph” visualizations there. Check it out after reading this post first. Introduction Brendan Gregg invented and popularized a way to profile & visualize program response time by sampling stack traces and using his FlameGraph concept & tools. This technique is a great way for visualizing metrics in nested hierarchies, what stack-based program execution uses under the hood for invoking and tracking function calls.

Generate Oracle SQL Monitoring Reports as HTML using SQL Developer v18.3 (no Flash needed)

Wed, 10 Oct 2018 17:00:52 -0400

The Oracle SQL Developer team has released version 18.3 of this tool. My favorite new feature in this version is its ability to save SQL Monitoring reports in HTML format! Adobe Flash is finally not needed for sharing graphical SQL Monitoring reports. Here’s an example output, it looks pretty nice and clean: The report is reasonably navigatable, has tooltips (useful for identifying wait classes for example) and you can also rearrange columns by dragging them left or right with your mouse.

Log file switch (checkpoint incomplete) wait events and LGWR waiting for checkpoint progress

Mon, 08 Oct 2018 00:00:08 -0400

I was recently involved in troubleshooting an interesting performance issue with some non-intuitive background process & wait event behavior. I reproduced the problem in my own custom-tailored environment (Oracle 18.3 on Linux), here’s the starting point: After starting a transaction-heavy benchmark, soon after the transaction rate drops and the database sessions are mostly waiting for some Configuration wait class events, instead of getting work done. Just looking into the high level wait class names is not enough, so let’s drill down with ashtop.

Oracle SQL Monitoring and Advanced ASH Usage Hacking Sessions!

Wed, 29 Aug 2018 14:25:38 -0400

It’s almost the beginning of September, time to get back to school, everyone! I recently decided to do my Oracle SQL Tuning and Performance & Troubleshooting training classes again. So I will also run some free hacking sessions for promoting these classes, but also for sharing cool new stuff with people who can’t attend the training and of course for fun too! The hacking sessions do not follow a formal training structure, have very few or no slides at all, just live demos of putting performance tools & techniques into use!

Oracle SQL Tuning training & Oracle Troubleshooting training and some Linux stuff too!

Mon, 20 Aug 2018 21:53:00 -0400

Update: Starting from 2022, I am delivering all my classes in a new format, via my new Learning Platform. Check out what are Learning Weeks and the new Continuous Learning method here: https://learn.tanelpoder.com More details about the class contents here: Advanced Oracle SQL Tuning training Practical Linux Performance & Application troubleshooting training Advanced Oracle Troubleshooting training Here are some example slides from the AST class for eye-candy :) Execution plan data flow Reading SQL Monitoring reports I hope you’ll find some of this stuff interesting and “See” you soon :-)

Webinar: Troubleshooting a Complex Oracle Performance Issue

Thu, 07 Jun 2018 09:26:57 +0300

I was recently involved in a discussion around why would a regular Oracle foreground session wait for the log file sequential read wait event. I then realized that I had never published my Hotsos Symposium 2014 talk about a relatively complex (treacherous?) Oracle performance issue involving multiple problems that “collide”. So I decided to run this presentation as a webinar next week + will demo some of my newer related scripts too!

My Performance & Troubleshooting scripts (TPT) for Oracle are now in GitHub and open sourced

Fri, 18 May 2018 20:13:22 +0000

I have uploaded my TPT-oracle scripts to GitHub and have formally open sourced them under Apache 2.0 license as well. This allows companies to embed this software in their toolsets and processes & distribute them without a worry from legal departments. The repository is here: https://github.com/tanelpoder/tpt-oracle Now you can “git clone” this repository once and just “git pull” every now and then to see what updates & fixes I have made.

Video: Oracle X$TRACE, Wait Event Internals and Background Process Communication

Thu, 25 Jan 2018 03:33:54 +0000

I have uploaded the the video of my Secret Hacking Session: Oracle X$TRACE, Wait Event Internals and Background Process Communication to my Oracle performance & troubleshooting Youtube channel. The slides are in Slideshare. Enjoy!

Secret Hacking Session: Oracle Background Process Communication, Exotic Wait Events and Some Tracing too

Thu, 11 Jan 2018 20:34:51 +0000

Update: I unexpectedly ended up falling ill and decided to reschedule this hacking session to January 24, 10am PST. No need to re-register if you already have done so. Sorry for the inconvenience. I will upload the video to Youtube after the event. Since I’m running my Advanced Oracle Troubleshooting Training in the end of this month, I’ll do one of my “secret” hacking sessions too for promotion and noise-making reasons next week!

Advanced Oracle Troubleshooting seminar in 2018!

Wed, 29 Nov 2017 22:24:06 +0000

A lot of people have asked me to do another run of my Oracle Troubleshooting training or at least get access to previous recordings – so I decided to geek out over the holiday period, update the material with latest stuff and run one more AOT class in 2018! The online training will take place on 29 January – 2 February 2018 (Part 1) & 26 February – 2 March 2018 (Part 2).

Apache Impala Internals Deep Dive with Tanel Poder + Gluent New World Training Month

Tue, 11 Jul 2017 16:51:21 +0000

We are running a “Gluent New World training month” in this July and have scheduled 3 webinars on following Wednesdays for this! The first webinar with Michael Rainey is going to cover modern alternatives to the traditional old-school “ETL on a RDBMS” approach for data integration and sharing. Then on the next Wednesday I will demonstrate some Apache Impala SQL engine’s internals, with commentary from an Oracle database geek’s angle (I plan to get pretty deep & technical).

GNW01: In-Memory Processing for Databases

Mon, 28 Mar 2016 05:39:25 +0000

This Gluent New World webinar is based on my RAM is the new disk and how to measure its performance article series: RAM is the new disk – Introduction RAM is the new disk – tools for measuring CPU efficiency and RAM access RAM is the new disk – Columnar vs Row-oriented data access test results I’m using the Oracle Database In-Memory option as an example here, but the same rules apply to other row & column stores as well.

My BIWA Summit Presentations

Tue, 26 Jan 2016 23:01:45 +0000

Here are the two BIWA Summit 2016 presentations I delivered today. The first one is a collection of high level thoughts (and opinions) of mine and the 2nd one is more technical: SQL in the Hybrid World from Tanel Poder SQL Monitoring in Oracle Database 12c from Tanel Poder

RAM is the new disk – and how to measure its performance – Part 3 – CPU Instructions & Cycles

Mon, 30 Nov 2015 06:45:58 +0000

If you haven’t read the previous parts of this series yet, here are the links: [ Part 1 | Part 2 ].

A Refresher

In the first part of this series I said that RAM access is the slow component of a modern in-memory database engine and for performance you’d want to reduce RAM access as much as possible. Reduced memory traffic thanks to the new columnar data formats is the most important enabler for the awesome In-Memory processing performance and SIMD is just icing on the cake.

In the second part I also showed how to measure the CPU efficiency of your (Oracle) process using a Linux perf stat command. How well your applications actually utilize your CPU execution units depends on many factors. The biggest factor is your process’es cache efficiency that depends on the CPU cache size and your application’s memory access patterns. Regardless of what the OS CPU accounting tools like top or vmstat may show you, your “100% busy” CPUs may actually spend a significant amount of their cycles internally idle, with a stalled pipeline, waiting for some event (like a memory line arrival from RAM) to happen.

Luckily there are plenty of tools for measuring what’s actually going on inside the CPUs, thanks to modern processors having CPU Performance Counters (CPC) built in to them.

A key derived metric for understanding CPU-efficiency is the IPC (instructions per cycle). Years ago people were actually talking about the inverse metric CPI (cycles per instruction) as on average it took more than one CPU cycle to complete an instruction’s execution (again, due to the abovementioned reasons like memory stalls). However, thanks to today’s superscalar processors with out-of-order execution on a modern CPU’s multiple execution units – and with large CPU caches – a well-optimized application can execute multiple instructions per a single CPU cycle, thus it’s more natural to use the IPC (instructions-per-cycle) metric. With IPC, higher is better.

My New Youtube Channel

Tue, 24 Nov 2015 04:30:28 +0000

I have created a new youtube channel – and have uploaded some videos there already! Bookmark & Subscribe here: https://www.youtube.com/tanelpoder More stuff is coming over the next weeks & months :-)

Troubleshooting Another Complex Performance Issue – Oracle direct path inserts and SEG$ contention

Wed, 11 Nov 2015 00:35:23 +0000

Here’s an updated presentation I first delivered at Hotsos Symposium 2015. It’s about lots of concurrent PX direct path insert ant CTAS statements that, when clashing with another bug/problem, caused various gc buffer busy waits and enq: TX – allocate ITL entry contention. This got amplified thanks to running this concurrent workload on 4 RAC nodes: When reviewing these slides, I see there’s quite a lot that needs to be said in addition to what’s on slides, so this might just mean a (Powerpoint) hacking session some day!

SQL Monitoring in Oracle Database 12c

Thu, 29 Oct 2015 18:53:08 +0000

Here’s my latest OOW presentation – SQL Monitoring in Oracle Database 12c: [direct link] You can download all my scripts from /files/

Advanced Oracle Troubleshooting v2.5 (with 12c stuff too)

Fri, 09 Oct 2015 07:58:08 +0000

It took a while (1.5 years since my last class – I’ve been busy!), but I am ready with my Advanced Oracle Troubleshooting training (version 2.5) that has plenty of updates, including some more modern DB kernel tracing & ASH stuff and of course Oracle 12c topics! The online training will take place on 16-20 November & **14-18 December 2015 **(Part 1 and Part 2). The latest TOC is below:

RAM is the new disk – and how to measure its performance – Part 2 – Tools

Mon, 21 Sep 2015 08:20:18 +0000

[ part 1 | part 2 | part 3 ]

In the previous article I explained that the main requirement for high-speed in-memory data scanning is column-oriented storage format for in-memory data. SIMD instruction processing is just icing on the cake. Let’s dig deeper. This is a long post, you’ve been warned.

Test Environment

I will cover full test results in the next article in this series. First, let’s look into the test setup, environment and what tools I used for peeking inside CPU hardware.

I was running the tests on a relatively old machine with 2 CPU sockets, with 6-core CPUs in each socket (2s12c24t):

$ egrep "MHz|^model name" /proc/cpuinfo | sort | uniq -c
     24 cpu MHz		: 2926.171
     24 model name	: Intel(R) Xeon(R) CPU           X5670  @ 2.93GHz

The CPUs support SSE4.2 SIMD extensions (but not the newer AVX stuff):

$ grep ^flags /proc/cpuinfo | egrep "avx|sse|popcnt" | sed 's/ /\n/g' | egrep "avx|sse|popcnt" | sort | uniq
popcnt
sse
sse2
sse4_1
sse4_2
ssse3

Even though the /proc/cpuinfo above shows the CPU clock frequency as 2.93GHz, these CPUs have Intel Turboboost feature that allows some cores run at up to 3.33GHz frequency when not all cores are fully busy and the CPUs aren’t too hot.

Indeed, the turbostat command below shows that the CPU core executing my Oracle process was running at 3.19GHz frequency:

# turbostat -p sleep 1
pk cor CPU    %c0  GHz  TSC SMI    %c1    %c3    %c6 CTMP   %pc3   %pc6
             6.43 3.02 2.93   0  93.57   0.00   0.00   59   0.00   0.00
 0   0   0   4.49 3.19 2.93   0  95.51   0.00   0.00   46   0.00   0.00
 0   1   1  10.05 3.19 2.93   0  89.95   0.00   0.00   50
 0   2   2   2.48 3.19 2.93   0  97.52   0.00   0.00   45
 0   8   3   2.05 3.19 2.93   0  97.95   0.00   0.00   44
 0   9   4   0.50 3.20 2.93   0  99.50   0.00   0.00   50
 0  10   5 100.00 3.19 2.93   0   0.00   0.00   0.00   59
 1   0   6   6.25 2.23 2.93   0  93.75   0.00   0.00   44   0.00   0.00
 1   1   7   3.93 2.04 2.93   0  96.07   0.00   0.00   43
 1   2   8   0.82 2.15 2.93   0  99.18   0.00   0.00   44
 1   8   9   0.41 2.48 2.93   0  99.59   0.00   0.00   41
 1   9  10   0.99 2.35 2.93   0  99.01   0.00   0.00   43
 1  10  11   0.76 2.36 2.93   0  99.24   0.00   0.00   44

I will come back to this CPU frequency turbo-boosting later when explaining some performance metrics.

I ran the experiments in Oct/Nov 2014, so used a relatively early Oracle 12.1.0.2.1 version with a bundle patch (19189240) for in-memory stuff.

The test was deliberately very simple as I was researching raw in-memory scanning and filtering speed and was not looking into join/aggregation performance. I was running the query below with different hints and parameters to change access path options:

SELECT COUNT(cust_valid) FROM customers_nopart c WHERE cust_id > 0

RAM is the new disk – and how to measure its performance – Part 1 – Introduction

Sun, 09 Aug 2015 23:26:55 +0000

[ part 1 | part 2 | part 3 ]

RAM is the new disk, at least in the In-Memory computing world.

No, I am not talking about Flash here, but Random Access Memory – RAM as in SDRAM. I’m by far not the first one to say it. Jim Gray wrote this in 2006: “Tape is dead, disk is tape, flash is disk, RAM locality is king” (presentation).

Also, I’m not going to talk about how RAM is faster than disk (everybody knows that), but in fact how RAM is the slow component of an in-memory processing engine.

I will use Oracle’s In-Memory column store and the hardware performance counters in modern CPUs for drilling down into the low-level hardware performance metrics about CPU efficiency and memory access.

But let’s first get started by looking a few years into past into the old-school disk IO and index based SQL performance bottlenecks :)

The Hybrid World is Coming

Mon, 29 Jun 2015 23:14:13 +0000

Here’s the video of E4 keynote we delivered together with Kerry Osborne a few weeks ago. It explains what we see is coming, at a high level, from long time Oracle database professionals’ viewpoint and using database terminology (as the E4 audience is all Oracle users like us). However, this change is not really about Oracle database world, it’s about a much wider shift in enterprise computing: modern Hadoop data lakes and clouds are here to stay.

Old ventures and new adventures

Fri, 19 Jun 2015 06:08:33 +0000

I have some news, two items actually. First, today (it’s still 18th June in California) is my blog’s 8th anniversary! I wrote my first blog post, about Advanced Oracle Troubleshooting, exactly 8 years ago, on 18th June 2007 and have written 229 blog posts since. I had started writing and accumulating my TPT script collection a couple of years earlier and now it has over 1000 files in it! And no, I don’t remember what all of them do and even why I had written them.

Advanced Oracle Troubleshooting Guide – Part 12: control file reads causing enq: SQ – contention waits?

Fri, 24 Apr 2015 23:23:32 +0000

Vishal Desai systematically troubleshooted an interesting case where the initial symptoms of the problem showed a spike of enq: SQ – contention waits, but he dug deeper – and found the root cause to be quite different. He followed the blockers of waiting sessions manually to reach the root cause – and also used my @ash/ash_wait_chains.sql and @ash/event_hist.sql scripts to extract the same information more conveniently (note that he had modified the scripts to take AWR snap_ids as time range parameters instead of the usual date/timestamp):

Sqlplus is my second home, part 8: Embedding multiple sqlplus arguments into one variable

Sun, 29 Mar 2015 21:23:51 +0000

I’ve updated some of my ASH scripts to use these 4 arguments in a standard way: What ASH columns to display (and aggregate by) Which ASH rows to use for the report (filter) Time range start Time range end So this means whenever I run ashtop (or dashtop) for example, I need to type in all 4 parameters. The example below would show top SQL_IDs only for user SOE sessions from last hour of ASH samples:

Oracle Exadata Performance: Latest Improvements and Less Known Features

Tue, 24 Mar 2015 14:57:51 +0000

Here are the slides of a presentation I did at the IOUG Virtual Exadata conference in February. I’m explaining the basics of some new Oracle 12c things related to Exadata, plus current latest cellsrv improvements like Columnar Flash Cache and IO skipping for Min/Max retrieval using Storage Indexes: Note that Christian Antognini and Roger MacNicol have written separate articles about some new features: http://antognini.ch/2015/03/exadata-storage-index-minmax-optimization/ https://blogs.oracle.com/smartscan-deep-dive/entry/examining_the_new_columnar_cache Enjoy!

Oracle In-Memory Column Store Internals – Part 1 – Which SIMD extensions are getting used?

Mon, 06 Oct 2014 05:51:30 +0000

This is the first entry in a series of random articles about some useful internals-to-know of the awesome Oracle Database In-Memory column store. I intend to write about Oracle’s IM stuff that’s not already covered somewhere else and also about some general CPU topics (that are well covered elsewhere, but not always so well known in the Oracle DBA/developer world).

Before going into further details, you might want to review the Part 0 of this series and also our recent Oracle Database In-Memory Option in Action presentation with some examples. And then read this doc by Intel if you want more info on how the SIMD registers and instructions get used.

There’s a lot of talk about the use of your CPUs’ SIMD vector processing capabilities in the Oracle inmemory module, let’s start by checking if it’s enabled in your database at all. We’ll look into Linux/Intel examples here.

About index range scans, disk re-reads and how your new car can go 600 miles per hour!

Wed, 17 Sep 2014 08:56:02 +0000

Despite the title, this is actually a technical post about Oracle, disk I/O and Exadata & Oracle In-Memory Database Option performance. Read on :)

If a car dealer tells you that this fancy new car on display goes 10 times (or 100 or 1000) faster than any of your previous ones, then either the salesman is lying or this new car is doing something radically different from all the old ones. You don’t just get orders of magnitude performance improvements by making small changes.

Perhaps the car bends space around it instead of moving – or perhaps it has a jet engine built on it (like the one below :-) :

Combining Bloom Filter Offloading and Storage Indexes on Exadata

Sun, 18 May 2014 04:15:18 +0000

Here’s a little known feature of Exadata – you can use a Bloom filter computed from a join column of a table to skip disk I/Os against another table it is joined to. This not the same as the Bloom filtering of the datablock contents in Exadata storage cells, but rather avoiding reading in some storage regions from the disks completely.

So, you can use storage indexes to skip I/Os against your large fact table, based on a bloom filter calculated from a small dimension table!

This is useful especially for dimensional star schemas, as your SQL statements might not have direct predicates on your large fact tables at all, all results will be determined by looking up relevant dimension records and then performing a hash join to the fact table (whether you should have some direct predicates against the fact tables, for performance reasons, is a separate topic for some other day :-)

Let me show an example using the SwingBench Order Entry schema. The first output is from Oracle 11.2.0.3 BP21 on Cellsrv 12.1.1.1.0:

What the heck are the /dev/shm/JOXSHM_EXT_x files on Linux?

Fri, 09 May 2014 20:12:01 +0000

There was an interesting question in Oracle-L about the JOXSHM_EXT_* files in /dev/shm directory on Linux. Basically something like this:

$ ls -l /dev/shm/* | head
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 /dev/shm/JOXSHM_EXT_0_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 /dev/shm/JOXSHM_EXT_100_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:16 /dev/shm/JOXSHM_EXT_101_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:23 /dev/shm/JOXSHM_EXT_102_LIN112_1409029
-rwxrwx--- 1 oracle dba 4096 Apr 18 10:23 /dev/shm/JOXSHM_EXT_103_LIN112_1409029
-rwxrwx--- 1 oracle dba 36864 Apr 18 10:23 /dev/shm/JOXSHM_EXT_104_LIN112_1409029
...

There are a few interesting MOS articles about these files and how/when to get rid of those (don’t remove any files before reading the notes!), but none of these articles explain why these JOXSHM (and PESHM) files are needed at all:

/dev/shm Filled Up With Files In Format JOXSHM_EXT_xxx_SID_xxx (Doc ID 752899.1)
Stale Native Code Files Are Being Cached with File Names Such as: JOXSHM_EXT*, PESHM_EXT*, PESLD* or SHMDJOXSHM_EXT* (Doc ID 1120143.1)
Ora-7445 [Ioc_pin_shared_executable_object()] (Doc ID 1316906.1)

Here’s an explanation, a bit more elaborated version of what I already posted in Oracle-L:

Oracle Memory Troubleshooting, Part 4: Drilling down into PGA memory usage with V$PROCESS_MEMORY_DETAIL

Wed, 26 Mar 2014 22:58:20 +0000

If you haven’t read them – here are the previous articles in Oracle memory troubleshooting series: Part 1, Part 2, Part 3.

Let’s say you have noticed that one of your Oracle processes is consuming a lot of private memory. The V$PROCESS has PGA_USED_MEM / PGA_ALLOC_MEM columns for this. Note that this view will tell you what Oracle thinks it’s using – how much of allocated/freed bytes it has kept track of. While this doesn’t usually tell you the true memory usage of a process, as other non-Oracle-heap allocation routines and the OS libraries may allocate (and leak) memory of their own, it’s a good starting point and usually enough.

Then, the V$PROCESS_MEMORY view would allow you to see a basic breakdown of that process’es memory usage – is it for SQL, PL/SQL, Java, unused (Freeable) or for “Other” reasons. You can use either the smem.sql or pmem.sql scripts for this (report v$process_memory for a SID or OS PID):

SQL> @smem 198
Display session 198 memory usage from v$process_memory....

       SID        PID    SERIAL# CATEGORY         ALLOCATED       USED MAX_ALLOCATED
---------- ---------- ---------- --------------- ---------- ---------- -------------
       198         43         17 Freeable           1572864          0
       198         43         17 Other              5481102                  5481102
       198         43         17 PL/SQL                2024        136          2024
       198         43         17 SQL              117805736  117717824     118834536

From the above output we see that this session has allocated over 100MB of private memory for “SQL” reasons. This normally means SQL workareas, so we can break this down further by querying V$SQL_WORKAREA_ACTIVE that shows us all currently in-use cursor workareas in the instance. I’m using a script wrka.sql for convenience – and listing only my SID-s workareas:

Slides of my previous presentations

Thu, 06 Mar 2014 13:57:10 +0000

Here are the slides of some of my previous presentations (that I haven’t made public yet, other than delivering these at conferences and training sessions):

Scripts and Tools That Make Your Life Easier and Help to Troubleshoot Better:

I delivered this presentation at the Hotsos Symposium Training Day in year 2010:

Troubleshooting Complex Performance Issues – Part1:

Where does the Exadata storage() predicate come from?

Wed, 05 Mar 2014 19:49:53 +0000

On Exadata (or when setting cell_offload_plan_display = always on non-Exadata) you may see the storage() predicate in addition to the usual access() and filter() predicates in an execution plan:

SQL> SELECT * FROM dual WHERE dummy = 'X';

D
-
X

Check the plan:

SQL> @x
Display execution plan for last statement for this session from library cache...

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------
SQL_ID  dtjs9v7q7zj1g, child number 0
-------------------------------------
SELECT * FROM dual WHERE dummy = 'X'

Plan hash value: 272002086

------------------------------------------------------------------------
| Id  | Operation                 | Name | E-Rows |E-Bytes| Cost (%CPU)|
------------------------------------------------------------------------
|   0 | SELECT STATEMENT          |      |        |       |     2 (100)|
|*  1 |  TABLE ACCESS STORAGE FULL| DUAL |      1 |     2 |     2   (0)|
------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   1 - storage("DUMMY"='X')
       filter("DUMMY"='X')

The access() and filter() predicates come from the corresponding ACCESS_PREDICATES and FILTER_PREDICATES columns in V$SQL_PLAN. But there’s no STORAGE_PREDICATES column there!

Oracle X$ tables – Part 1 – Where do they get their data from?

Fri, 10 Jan 2014 19:38:18 +0000

It’s long-time public knowledge that X$ fixed tables in Oracle are just “windows” into Oracle’s memory. So whenever you query an X$ table, the FIXED TABLE rowsource function in your SQL execution plan will just read some memory structure, parse its output and show you the results in tabular form. This is correct, but not the whole truth.

Check this example. Let’s query the X$KSUSE table, which is used by V$SESSION:

SQL> SELECT addr, indx, ksuudnam FROM x$ksuse WHERE rownum <= 5;

ADDR           INDX KSUUDNAM
-------- ---------- ------------------------------
391513C4          1 SYS
3914E710          2 SYS
3914BA5C          3 SYS
39148DA8          4 SYS
391460F4          5 SYS

Now let’s check in which Oracle memory region this memory address resides (SGA, PGA, UGA etc). I’m using my script fcha for this (Find CHunk Address). You should probably not run this script in busy production systems as it uses the potentially dangerous X$KSMSP fixed table:

SQL> @fcha 391513C4
Find in which heap (UGA, PGA or Shared Pool) the memory address 391513C4 resides...

WARNING!!! This script will query X$KSMSP, which will cause heavy shared pool latch contention
in systems under load and with large shared pool. This may even completely hang
your instance until the query has finished! You probably do not want to run this in production!

Press ENTER to continue, CTRL+C to cancel...

LOC KSMCHPTR   KSMCHIDX   KSMCHDUR KSMCHCOM           KSMCHSIZ KSMCHCLS   KSMCHTYP KSMCHPAR
--- -------- ---------- ---------- ---------------- ---------- -------- ---------- --------
SGA 39034000          1          1 permanent memor     3977316 perm              0 00

SQL>

Ok, these X$KSUSE (V$SESSION) records reside in a permanent allocation in SGA and my X$ query apparently just parsed & presented the information from there.

Now, let’s query something else, for example the “Soviet Union” view X$KCCCP:

cell flash cache read hits vs. cell writes to flash cache statistics on Exadata

Wed, 04 Dec 2013 11:57:09 +0000

When the Smart Flash Cache was introduced in Exadata, it was caching reads only. So there were only read “optimization” statistics like cell flash cache read hits and physical read requests/bytes optimized in V$SESSTAT and V$SYSSTAT (the former accounted for the read IO requests that got its data from the flash cache and the latter ones accounted the disk IOs avoided both thanks to the flash cache and storage indexes). So if you wanted to measure the benefit of flash cache only, you’d have to use the cell flash cache read hits metric.

This all was fine until you enabled the Write-Back flash cache in a newer version of cellsrv. We still had only the “read hits” statistic in the V$ views! And when investigating it closer, both the read hits and write hits were accumulated in the same read hits statistic! (I can’t reproduce this on our patched 11.2.0.3 with latest cellsrv anymore, but it was definitely the behavior earlier, as I demoed it in various places).

Side-note: This is likely because it’s not so easy to just add more statistics to Oracle code within a single small patch. The statistic counters are referenced by other modules using macros with their direct numeric IDs (and memory offsets to v$sesstat array) and the IDs & addresses would change when more statistics get added. So, you can pretty much add new statistic counters only with new full patchsets, like 11.2.0**.4**. It’s the same with instance parameters by the way, that’s why the “spare” statistics and spare parameters exist, they’re placeholders for temporary use, until the new parameter or statistic gets added permanently with a full patchset update.

So, this is probably the reason why both the flash cache read and write hits got initially accumulated under the cell flash cache read hits statistic, but later on this seemed to get “fixed”, so that the read hits only showed read hits and the flash write hits were not accounted anywhere. You can test this easily by measuring your DBWR’s v$sesstat metrics with snapper for example, if you get way more cell flash cache read hits than physical read total IO requests, then you’re probably accumulating both read and write hits in the same metric.

Let’s look into a few different database versions:

Hard Drive Predictive Failures on Linux and Exadata

Fri, 29 Nov 2013 12:50:00 +0000

This post also applies to non-Exadata systems as hard drives work the same way in other storage arrays too – just the commands you would use for extracting the disk-level metrics would be different. Scroll down to smartctl if you wan’t to skip the Oracle stuff and get straight to the Linux disk diagnosis commands.

I just noticed that one of our Exadatas had a disk put into “predictive failure” mode and thought to show how to measure why the disk is in that mode (as opposed to just replacing it without really understanding the issue ;-)

SQL> @exadata/cellpd
Show Exadata cell versions from V$CELL_CONFIG....

DISKTYPE             CELLNAME             STATUS                 TOTAL_GB     AVG_GB  NUM_DISKS   PREDFAIL   POORPERF WTCACHEPROB   PEERFAIL   CRITICAL
-------------------- -------------------- -------------------- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------
FlashDisk            192.168.12.3         normal                      183         23          8
FlashDisk            192.168.12.3         not present                 183         23          8                     3
FlashDisk            192.168.12.4         normal                      366         23         16
FlashDisk            192.168.12.5         normal                      366         23         16
HardDisk             192.168.12.3         normal                    20489       1863         11
HardDisk             192.168.12.3         warning - predictive       1863       1863          1          1
HardDisk             192.168.12.4         normal                    22352       1863         12
HardDisk             192.168.12.5         normal                    22352       1863         12

So, one of the disks in storage cell with IP 192.168.12.3 has been put into predictive failure mode. Let’s find out why!

When do Oracle Parallel Execution Slaves issue buffered physical reads – Part 2?

Wed, 27 Nov 2013 11:32:36 +0000

In the previous post about in-memory parallel execution I described in which cases the in-mem PX can kick in for your parallel queries. A few years ago (around Oracle 11.2.0.2 and Exadata X2 release time) I was helping a customer with their migration to Exadata X2. Many of the queries ran way slower on Exadata compared to their old HP Superdome. The Exadata system was configured according to the Oracle’s “best practices”, that included setting the parallel_degree_policy = AUTO.

When do Oracle Parallel Execution Slaves issue buffered physical reads – Part 1?

Wed, 27 Nov 2013 10:32:38 +0000

This post applies both to non-Exadata and Exadata systems.

Before Oracle 11.2 came out, it was true to say that Oracle Parallel Execution slaves always do direct path reads (bypassing buffer cache) when doing full segment scans. This should not be taken simplistically though. Even when you were doing full table scans, then yes the scanning was done with direct path multiblock reads – but if you had to visit other, additional blocks out of the scanning sequence, then these extra IOs were done with regular buffered reads. For example, next row piece fetching of chained rows or or undo block access for CR reads was done with buffered single block reads, or even buffered multiblock reads, if some form of prefetching kicked in.

In addition to that, random table/index accesses like index range scans and the following table block fetches are always done in a buffered way both in serial and parallel execution cases.

Starting from Oracle 11.2 though, Oracle parallel execution slaves can also do the parallel full segment scans via the buffer cache. The feature is called In-Memory Parallel Execution, not to be confused with the Oracle 12c upcoming In-Memory Option (which gives you a columnar, compressed, in-memory cache of your on-disk data).

Diagnosing buffer busy waits with the ash_wait_chains.sql script (v0.2)

Thu, 07 Nov 2013 01:54:01 +0000

In my previous post ( Advanced Oracle Troubleshooting Guide – Part 11: Complex Wait Chain Signature Analysis with ash_wait_chains.sql ) I introduced an experimental script for analysing performance from “top ASH wait chains” perspective. The early version (0.1) of the script didn’t have the ability to select a specific user (session), SQL or module/action performance data for analysis. I just hadn’t figured out how to write the SQL for this (as the blocking sessions could come from any user).

SGA bigger than the amount of HugePages configured (Linux – 11.2.0.3)

Fri, 25 Oct 2013 13:23:24 +0000

I just learned something new yesterday when demoing large page use on Linux during my AOT seminar. I had 512 x 2MB hugepages configured in Linux ( 1024 MB ). So I set the USE_LARGE_PAGES = TRUE (it actually is the default anyway in 11.2.0.2+). This allows the use of large pages (it doesn’t force, the ONLY option would force the use of hugepages, otherwise the instance wouldn’t start up). Anyway, the previous behavior with hugepages was, that if Oracle was not able to allocate the entire SGA from the hugepages area, it would silently allocate _the entire SGA _from small pages.

Why doesn’t ALTER SYSTEM SET EVENTS set the events or tracing immediately?

Mon, 07 Oct 2013 15:23:22 +0000

I received a question about ALTER SYSTEM in the comments section of another blog post recently.

The question was that while ALTER SESSION SET EVENTS ‘10046 … ‘ enabled the SQL Trace for the current session immediately, ALTER SYSTEM on the other hand didn’t seem to do anything at all for other sessions in the instance.

There’s an important difference in the behavior of ALTER SYSTEM when changing paramters vs. setting events.

For example, ALTER SYSTEM SET optimizer_mode = CHOOSE would change the value of this parameter immediately, for:

Your own session
All new sessions that will log in will pick up the new parameter value
All other existing sessions

However, when you issue an ALTER SYSTEM SET EVENTS ‘10046 TRACE NAME CONTEXT FOREVER, LEVEL 12’, the event changes in only #1 and #2 will happen:

Your own session
All new sessions that will log in will pick up the new event settings

This means that the existing, already logged in sessions, will not pick up any of the events set via ALTER SYSTEM!

Update 1: Note that since Oracle 11g there’s a parameteter “_evt_system_event_propagation” (default TRUE) and it makes Oracle to propagate the “ALTER SYSTEM” events into existing sessions too. So this problem described above applies only up to Oracle 10.2.

Update 2: Oracle 12.2 seems to have a bug that prevents some events (at least one ALTER SYSTEM SET sql_trace example with a SQL_ID qualifier) from being propagated correctly to other database sessions (link)

Update 3: Apparently in Oracle 18.3 the abovementioned bug is fixed.

This hopefully explains why sometimes the debug events don’t seem to work. But more importantly, this also means that when you disable an event (by setting it to “OFF” or to level 0) with ALTER SYSTEM, it does not affect the existing sessions who have this event enabled! So, you think you’re turning the tracing off for all sessions and go home, but really some sessions keep on tracing – until the filesystem is full (and you’ll get a phone call at 3am).

Advanced Oracle Troubleshooting Guide – Part 11: Complex Wait Chain Signature Analysis with ash_wait_chains.sql

Thu, 12 Sep 2013 01:23:35 +0000

Here’s a treat for the hard-core Oracle performance geeks out there – I’m releasing a cool, but still experimental script for ASH (or poor-man’s ASH)-based wait event analysis, which should add a whole new dimension into ASH based performance analysis. It doesn’t replace any of the existing ASH analysis techniques, but should bring the relationships between Oracle sessions in complex wait chains out to bright daylight much easier than before.

You all are familiar with the AWR/Statspack timed event summary below:

Similar breakdown can be gotten by just aggregating ASH samples by the wait event:

SQL> @ash/dashtop session_state,event 1=1 "TIMESTAMP'2013-09-09 21:00:00'" "TIMESTAMP'2013-09-09 22:00:00'"

%This  SESSION EVENT                                                            TotalSeconds        CPU   User I/O Application Concurrency     Commit Configuration    Cluster       Idle    Network System I/O  Scheduler Administrative   Queueing      Other MIN(SAMPLE_TIME)                                                            MAX(SAMPLE_TIME)
------ ------- ---------------------------------------------------------------- ------------ ---------- ---------- ----------- ----------- ---------- ------------- ---------- ---------- ---------- ---------- ---------- -------------- ---------- ---------- --------------------------------------------------------------------------- ---------------------------------------------------------------------------
  68%  ON CPU                                                                          25610      25610          0           0           0          0             0          0          0          0          0          0              0          0          0 09-SEP-13 09.00.01.468 PM                                                   09-SEP-13 09.59.58.059 PM
  14%  WAITING SQL*Net more data from client                                            5380          0          0           0           0          0             0          0          0       5380          0          0              0          0          0 09-SEP-13 09.00.01.468 PM                                                   09-SEP-13 09.59.58.059 PM
   6%  WAITING enq: HW - contention                                                     2260          0          0           0           0          0          2260          0          0          0          0          0              0          0          0 09-SEP-13 09.04.41.893 PM                                                   09-SEP-13 09.56.07.626 PM
   3%  WAITING log file parallel write                                                  1090          0          0           0           0          0             0          0          0          0       1090          0              0          0          0 09-SEP-13 09.00.11.478 PM                                                   09-SEP-13 09.59.58.059 PM
   2%  WAITING db file parallel write                                                    730          0          0           0           0          0             0          0          0          0        730          0              0          0          0 09-SEP-13 09.01.11.568 PM                                                   09-SEP-13 09.59.48.049 PM
   2%  WAITING enq: TX - contention                                                      600          0          0           0           0          0             0          0          0          0          0          0              0          0        600 09-SEP-13 09.04.41.893 PM                                                   09-SEP-13 09.48.16.695 PM
   1%  WAITING buffer busy waits                                                         560          0          0           0         560          0             0          0          0          0          0          0              0          0          0 09-SEP-13 09.10.02.492 PM                                                   09-SEP-13 09.56.07.626 PM
   1%  WAITING log file switch completion                                                420          0          0           0           0          0           420          0          0          0          0          0              0          0          0 09-SEP-13 09.47.16.562 PM                                                   09-SEP-13 09.47.16.562 PM
   1%  WAITING latch: redo allocation                                                    330          0          0           0           0          0             0          0          0          0          0          0              0          0        330 09-SEP-13 09.04.41.893 PM                                                   09-SEP-13 09.53.27.307 PM
...

The abovementioned output has one shortcoming in a multiuser (database) system – not all wait events are simple, where a session waits for OS to complete some self-contained operation (like an IO request). Often a session waits for another session (who holds some lock) or some background process who needs to complete some task before our session can continue. That other session may wait for a yet another session due to some other lock. The other session itself waits for a yet another one, thanks to some buffer pin (buffer busy wait). The session who holds the buffer pin, may itself be waiting for LGWR, who may in turn wait for DBWR etc… You get the point – sometimes we have a bunch of sessions, waiting for each other in a chain.

The V$WAIT_CHAINS view introduced in Oracle 11g is capable of showing such chains of waiting sessions – however it is designed to diagnose relatively long-lasting hangs, not performance problems and short (but non-trivial) contention. Usually the DIAG process, who’s responsible for walking through the chains and populating V$WAIT_CHAINS, doesn’t kick in after a few seconds of ongoing session waits and the V$WAIT_CHAINS view may be mostly empty – so we need something different for performance analysis.

Oracle Performance & Troubleshooting Online Seminars in 2013

Thu, 05 Sep 2013 13:45:07 +0000

In case you haven’t noticed, I will be delivering my Advanced Oracle Troubleshooting and Advanced Oracle Exadata Performance: Troubleshooting and Optimization classes again in Oct/Nov 2013 (AOT) and December 2013 (Exadata). I have streteched the Exadata class to 5-half days as 4 half-days wasn’t nearly enough to deliver the amount of details in the material (and I think it’s still going to be a pretty intensive pace). And that’s all for this year (I will write about conferences and other public appearances in a separate post).

Scalar Subqueries in Oracle SQL WHERE clauses (and a little bit of Exadata stuff too)

Thu, 22 Aug 2013 21:03:25 +0000

My previous post was about Oracle 12c SQL Scalar Subquery transformations. Actually I need to clarify its scope a bit: the previous post was about scalar subqueries inside a SELECT projection list only (meaning that for populating a field in the query resultset, a subquery gets executed once for each row returned back to the caller, instead of returning a “real” column value passed up from a child rowsource).

I did not cover an other use case in my previous post – it is possible to use scalar subqueries also in the WHERE clause, for filtering the resultset, so let’s see what happens in this case too!

Note that the tests below are ran on an Oracle 11.2.0.3 database (not 12c as in the previous post), because I want to add a few Exadata details to this post – and as of now, 18th August 2013, Smart Scans don’t work with Oracle 12c on Exadata. This will of course change once the first Oracle 12c patchset will be released, but this will probably happen somewhere in the next year.

So, let’s look into the following simple query. The bold red part is the scalar subquery (well, as long as it returns 0 or 1 rows, if it returns more, you’ll get an error during query execution). I’m searching for “objects” from a test_objects_100m table (with 100 Million rows in it), but I only want to process the rows where the object’s owner name is whatever the subquery on test_users table returns. I have also disabled Smart Scans for this query so that the database would behave more like a regular non-Exadata DB for now:

SELECT /*+ MONITOR OPT_PARAM('cell_offload_processing', 'false') */
    SUM(LENGTH(object_name)) + SUM(LENGTH(object_type)) + SUM(LENGTH(owner))
FROM
    test_objects_100m o
WHERE
    o.owner = (SELECT u.username FROM test_users u WHERE user_id = 13)
/

Note the equals (=) sign above, I’m simply looking for a single, noncorrelated value from the subquery – it’s not a more complex (and unpredictable!) IN or EXISTS subquery. Let’s see the execution plan, pay attention to the the table names and execution order below:

------------------------------------------------------------------------------------------------------------
| Id  | Operation                              | Name              | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                       |                   |     1 |    33 |   405K  (1)| 00:52:52 |
|   1 |  SORT AGGREGATE                        |                   |     1 |    33 |            |          |
|*  2 |   TABLE ACCESS STORAGE FULL            | TEST_OBJECTS_100M |  7692K|   242M|   405K  (1)| 00:52:52 |
|*  3 |    TABLE ACCESS STORAGE FULL FIRST ROWS| TEST_USERS        |     1 |    12 |     3   (0)| 00:00:01 |
------------------------------------------------------------------------------------------------------------

   2 - filter("O"."OWNER"= (SELECT "U"."USERNAME" FROM "TEST_USERS" "U" WHERE "USER_ID"=13))
   3 - filter("USER_ID"=13)

That sure is a weird-looking execution plan, right?

ShowMOS: How to get rid of the annoying “The Page has Expired” dialog in My Oracle Support

Mon, 19 Aug 2013 12:00:58 +0000

So, how many of you do hate the dialog below? Good news – there is a fix! (or well, a hack around it ;) Before showing the fix, you can vote & give your opinion here: Do you love or hate the MOS “page expired” dialog? The fix is actually super-simple. The page expiration dialog that grays out the browser screen is just a HTML DIV with ID DhtmlZOrderManagerLayerContainer, overlaying the useful content.

Oracle 12c: Scalar Subquery Unnesting transformation

Tue, 13 Aug 2013 14:55:27 +0000

I promised to write about Oracle 12c new features quite a while ago (when 12c got officially released), but I was actually on (a long) vacation then and so many cool 12c-related white-papers and blog entries started popping up so I took it easy for a while. I plan to be focusing on the less known low-level internal details anyway as you see from this blog entry.

As far as I can remember, Oracle has been able to unnest regular subqueries since 8i and merge views since Oracle 8.0.

First, a little terminology:

Update: I have changed the terminology section below a bit, thanks to Jason Bucata’s correction. A scalar subquery sure can also be used in the WHERE clause (as you can see in the comments). So, I clarified below that this blog post is comparing the “Scalar Subqueries in SELECT projection list” to “Regular non-scalar Subqueries in WHERE clause”. I also wrote a Part 2 to explain the scalar subqueries in WHERE clause.

A “regular” subquery in Oracle terminology is a query block which doesn’t return rows back to the select projection list, but is only used in the WHERE clause for determining whether to return any rows from the parent query block, based on your specified conditions, like WHERE v IN (SELECT x FROM q) or WHERE EXISTS (SELECT x FROM inner WHERE inner.v=outer.v) etc.

The subquery can return zero to many rows for the parent query block for evaluation.

A scalar subquery is a subquery which can only return a single value (single row, single column) back to the parent block. It can be used both in the WHERE clause of the parent query or right in the SELECT list of the parent query instead of some column name. (In this post I am discussing only the case of a scalar subquery in SELECT clause). Whatever the scalar subquery returns, will be put in the “fake” column in the query result-set. If it returns more than one row, you will get the ORA-01427: single-row subquery returns more than one row error, if it returns no rows for the given lookup, the result will be NULL. An example is below.

I crafted a very simple SQL with a scalar subquery for demo purposes (the tables are just copies of all_users and all_objects views):

SELECT
    u.username
  , (SELECT MAX(created) FROM test_objects o WHERE o.owner = u.username)
FROM
    test_users u
WHERE
    username LIKE 'S%'

Now, why would you want to write the query this way is a different story. I think it’s actually pretty rare when you need to use a scalar subquery, you usually can get away with an outer join. I have used scalar subqueries for populating some return values in cases where adding a yet another (outer) join to the query would complicate the query too much for my brain (as there are some limitations how you can arrange outer joins). I have only done this when I know that the query result-set (on which the scalar subquery is executed once for every row returned, unless the subquery caching kicks in!) will only return a handful of rows and the extra effort of running the scalar subquery once for each row is acceptable.

Nevertheless, non-experienced SQL developers (who come from the procedural coding world) write lots of scalar subqueries, even up to the point of having every single column populated by a scalar subquery! And this can be a serious problem as this breaks the query into separate non-mergeable chunks, which means that the CBO isn’t as free to move things around – resulting in suboptimal plans.

So, starting from Oracle 12c (and maybe even 11.2.0.4?), the CBO transformation engine can unnest some types of the scalar subqueries and convert these to outer joins internally.

Oracle Database 12c R1 (12.1.0.1.0) is finally released!

Tue, 25 Jun 2013 18:39:46 +0000

You can download it from http://edelivery.oracle.com And now I (and other beta testers) can finally start writing about the new cool features in it! :) Looks like only the Linux x86-64, Solaris x86-64 + SPARC ports are available first (as usual). (just a screenshot below, you’ll need to go to http://edelivery.oracle.com to download the files) Update: Also the Oracle 12c Documentation is now available in the OTN website.

Getting the Most Out of ASH online seminar

Wed, 05 Jun 2013 21:10:35 +0000

Just a little reminder – next week (10-11th June) I’ll be delivering my last training session before autumn – a short 1-day (2 x 0.5 days actually) seminar about Getting the Most Out of Oracle’s Active Session History. In the future it will act as sort of a prequel (or preparation) for my Advanced Oracle Troubleshooting class, as the latter one deliberately goes very deep. The ASH seminar’s 1st half is actually mostly about the GUI way of troubleshooting the usual performance problems (EM/Grid Control) and the 2nd half is about all my ASH scripts for diagnosing more complex stuff.

Forcing Smart Scans on Exadata – is the _serial_direct_read parameter safe to use in production?

Wed, 29 May 2013 12:38:14 +0000

One of the most common Exadata performance problems I see is that the direct path reads (and thus also Smart Scans) don’t sometimes kick in when running full scans in serial sessions. This is because in Oracle 11g onwards, the serial full segment scan IO path decision is done dynamically, at runtime, for every SQL execution – and for every segment (partition) separately. Whether you get a direct path read & smart scan, depends on the current buffer cache size, how big segment you’re about to scan and how much of that segment is actually cached at the moment. Note that the automatic IO path decision for index fast full scans is slightly different from table scans.

This dynamic decision unfortunately can cause unexpected surprises and variance in your report/batch job runtimes. Additionally, it looks like the SELECT part of your UPDATE/DELETE statements (the select part finds the rows to update/delete) does not ever automatically get direct path read/smart scan chosen – by design! So, when your SELECT statement may use smart scan and be really fast, the same select operation in an INSERT SELECT (or UPDATE/DELETE) context will not end up using smart scans by default. There’s even a bug explaining that – closed as “not a bug” (Exadata Smartscan Is Not Being Used On Insert As Select[Article ID 1348116.1]).

To work around these problems and force a direct path read/smart scan, you can either:

Run your query in parallel as parallel full segment scans will use direct path reads, unless your parallel_degree_policy = AUTO, then you may still get buffered reads thanks to the dynamic in-memory parallel execution decision of Oracle 11.2
Run your query in serial, but force the serial direct path reads by setting _serial_direct_read = TRUE (or ALWAYS in 11.2.0.2+)

Here are the valid options for this parameter in 11.2.0.2+

Debugger Dangers – Part 2

Mon, 27 May 2013 17:31:45 +0000

About 5 years ago I wrote about the risks that connecting to Oracle processes via debuggers may cause and what are (in my opinion) the safer and less safer options for taking stack samples from running Oracle processes.

In the end of that article I listed different options for getting a stack traces and whether they were safe or not.

For example, ORADEBUG-based process stack traces (DUMP ERRORSTACK, SHORT_STACK and event the process/system state dumps (at level 256 or higher) are not 100% safe – because they alter the execution path of the process they attached to. Your process may crash or get some error if you hit a bug (of course once you patch/fix the bug, you’ll be fine again – until you may hit the next bug).

An example bug is this:

Bug 15677306 : SUNBT6994922 ORACLE LOGWRITER HARD HANG WHEN SIGUSR INTERRUPTS LIBAIO

LGWR may hang when SIGUSR interrupts libaio (that’s what LGWR uses for asynchronous IO). And if LGWR is hung – your whole instance will be hung pretty soon. But what does the SIGUSR signal have to do with Oracle? Well, that’s exactly the signal what ORADEBUG sends to the process it has attached to, to notify it about some debug/dump work it needs to do. And apparently receiving or handling such a signal (jumping into some stack dump function), when being in libaio codepath causes an issue.

List Exadata Storage Cell disk summary with cellpd.sql and cellpdx.sql scripts

Fri, 24 May 2013 20:05:50 +0000

In the previous post I explained how to list Exadata disk layout and topology details with the exadisktopo scripts, in this post I’ll introduce one celldisk overview script, which I use to quickly see the celldisk configuration, specs and error statuses. The cellpd.sql script (Cell Physical Disk) will show the following output:

SQL> @exadata/cellpd.sql
Show Exadata cell versions from V$CELL_CONFIG....

DISKTYPE             CELLNAME             STATUS                 TOTAL_GB     AVG_GB  NUM_DISKS   PREDFAIL   POORPERF WTCACHEPROB   PEERFAIL   CRITICAL
-------------------- -------------------- -------------------- ---------- ---------- ---------- ---------- ---------- ----------- ---------- ----------
FlashDisk            192.168.12.3         normal                      183         23          8
FlashDisk            192.168.12.3         not present                 183         23          8                     3
FlashDisk            192.168.12.4         normal                      366         23         16
FlashDisk            192.168.12.5         normal                      366         23         16
HardDisk             192.168.12.3         normal                    22352       1863         12
HardDisk             192.168.12.4         normal                    22352       1863         12
HardDisk             192.168.12.5         normal                    22352       1863         12

Above you see that some of my Flash Disks are missing and before they went completely missing (because the Oracle Ninja removed them :) these disks indicated the “poor performance” status.

The cellpdx.sql script will display extended info for each celldisk, including disk model, firmware version, serial number and various error metrics. Warning, lots of wide output coming again:

List Exadata Disk Layout and Topology with the exadisktopo scripts

Fri, 24 May 2013 19:42:29 +0000

Here are two more Exadata scripts for listing the end-to-end ASM<->Exadata disk topology from V$ASM_ views and from V$CELL_CONFIG. These scripts see both the ASM level layout and the storage cell-level disk topology.

The exadisktopo.sql script shows all disks starting from the ASM diskgroup layer, going deeper and deeper all the way to the OS disk device level in the storage cells. It uses outer joins, so will show celldisks even if there are no corresponding grid disks allocated on them (or if there are no ASM disks using them). It also shows the Flash cards used as flash cache, thus there are no ASM disks on them usually.

The output should be self-explanatory, you just read the data from left to right (from ASM diskgroup to ASM disk to Grid Disk level etc), the more rightwards you scroll, the “deeper” you go in the Exadata IO stack layers. Warning, lots of wide output coming :)

V$CELL_THREAD_HISTORY – “ASH” for Exadata Storage Cells

Thu, 23 May 2013 20:57:33 +0000

Did you know that there’s something like Active Session History also in the Exadata storage cells? ;-)

The V$CELL_THREAD_HISTORY view is somewhat like V$ACTIVE_SESSION_HISTORY, but it’s measuring thread activity in the Exadata Storage Cells:

SQL> @desc v$cell_thread_history
           Name                            Null?    Type
           ------------------------------- -------- ----------------------------
    1      CELL_NAME                                VARCHAR2(1024)
    2      SNAPSHOT_ID                              NUMBER
    3      SNAPSHOT_TIME                            DATE
    4      THREAD_ID                                NUMBER
    5      JOB_TYPE                                 VARCHAR2(32)
    6      WAIT_STATE                               VARCHAR2(32)
    7      WAIT_OBJECT_NAME                         VARCHAR2(32)
    8      SQL_ID                                   VARCHAR2(13)
    9      DATABASE_ID                              NUMBER
   10      INSTANCE_ID                              NUMBER
   11      SESSION_ID                               NUMBER
   12      SESSION_SERIAL_NUM                       NUMBER

It keeps about 10 minutes worth of samples of Exadata Storage Cell thread activity:

SQL> @minmax snapshot_time v$cell_thread_history
Show min/max (low/high) values in column "snapshot_time" of table v$cell_thread_history...

MIN(SNAPSHOT_TIME MAX(SNAPSHOT_TIME
----------------- -----------------
20130419 14:42:15 20130419 14:52:54

Note that it’s not the V$ view or the database instance which stores this array – it’s the storage cells themselves. If you query the v$cell_thread_history view, your Oracle database session is going to “gather” this instrumentation data from all the required cells and present it to you, that’s why the “cell statistics gather” wait event shows up:

PARSING IN CURSOR #140596385017248 len=42 dep=0 uid=0 oct=3 lid=0 tim=1366404896817011 hv=4063158547 ad='19e452578' sqlid='63awy1gg
t2xs8m'
select count(*) from v$cell_thread_history
END OF STMT
PARSE #140596385017248:c=0,e=115,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4272803188,tim=1366404896817010
EXEC #140596385017248:c=0,e=27,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=1,plh=4272803188,tim=1366404896817127
WAIT #140596385017248: nam='SQL*Net message to client' ela= 1 driver id=1413697536 #bytes=1 p3=0 obj#=40 tim=1366404896817152
WAIT #140596385017248: nam='cell statistics gather' ela= 283 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896818846
WAIT #140596385017248: nam='cell statistics gather' ela= 352 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896819317
WAIT #140596385017248: nam='cell statistics gather' ela= 376 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896820929
WAIT #140596385017248: nam='cell statistics gather' ela= 326 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896822198
WAIT #140596385017248: nam='cell statistics gather' ela= 580 cellhash#=0 p2=0 p3=0 obj#=40 tim=1366404896823620
...

Now, how frequently do the cells sample their “ASH” data:

SQL> SELECT DISTINCT (snapshot_time - LAG(snapshot_time) 
                      OVER (ORDER BY snapshot_time)) * 86400 lag_seconds
     FROM (SELECT distinct snapshot_time 
           FROM v$cell_thread_history WHERE cell_name = '192.168.12.3');

LAG_SECONDS
-----------

          1

Looks like the sampling is done exactly once per second!

So, great, what can we do with this data?

Here’s an Exadata performance script (cth.sql) which uses V$CELL_THREAD_HISTORY to dig into cell activity from the database layer.

Understanding what a hint affects using the V$SQL_FEATURE views

Mon, 01 Apr 2013 17:17:07 +0000

You may have used the Oracle 11g V$SQL_HINT view already – it displays all the valid hints (both documented and undocumented ones) available in your Oracle version, for example:

SQL> @hint merge

NAME                                VERSION                   VERSION_OUTLINE           INVERSE
----------------------------------- ------------------------- ------------------------- ----------------------------------------------------------------
MERGE_CONST_ON                      8.0.0
MERGE_AJ                            8.1.0                     8.1.7
MERGE_SJ                            8.1.0                     8.1.7
MV_MERGE                            9.0.0
MERGE                               8.1.0                     10.1.0                    NO_MERGE
NO_MERGE                            8.0.0                     10.1.0                    MERGE
USE_MERGE_CARTESIAN                 11.1.0.6                  11.1.0.6
USE_MERGE                           8.1.0                     8.1.7                     NO_USE_MERGE
NO_USE_MERGE                        10.1.0.3                  10.1.0.3                  USE_MERGE

But there’s more, (semi)undocumented views like V$SQL_FEATURE and V$SQL_FEATURE_HIERARCHY do give us more information about what these hints relate to. For example, if you have ever wondered why is there a MERGE hint and then also a USE_MERGE hint, you can check what do these hints control using my hinth.sql (Hint Hierarchy) script:

SQL> @hinth MERGE
Display Hint feature hierarchy for hints like MERGE

NAME                               PATH
---------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
MERGE                              ALL -> COMPILATION -> CBO -> CBQT -> CVM
MERGE                              ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> CVM
MERGE                              ALL -> COMPILATION -> TRANSFORMATION -> HEURISTIC -> CVM

So, the MERGE hints seem to affect the CBO’s query transformation code – (CBQT means Cost-Based Query Transformation and CVM means Complex View Merging, but more about that later).

SQL> @hinth USE_MERGE
Display Hint feature hierarchy for hints like USE_MERGE

NAME                               PATH
---------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
USE_MERGE                          ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_MERGE

And the USE_MERGE hint is about controlling the use of a join method – the sort-merge join.

Let’s list all hints having NL in them:

SQL> @hinth %NL%
Display Hint feature hierarchy for hints like %NL%

NAME                               PATH
---------------------------------- ------------------------------------------------------------------------------------------------------------------------------------------------------
INLINE_XMLTYPE_NT                  ALL
NL_SJ                              ALL -> COMPILATION -> CBO
NL_AJ                              ALL -> COMPILATION -> CBO
NO_TABLE_LOOKUP_BY_NL              ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                 ALL -> COMPILATION -> CBO -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_USE_NL                          ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL
USE_NL                             ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL
USE_NL_WITH_INDEX                  ALL -> COMPILATION -> CBO -> JOIN_METHOD -> USE_NL_WITH_INDEX
NO_TABLE_LOOKUP_BY_NL              ALL -> COMPILATION -> CBO -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                 ALL -> COMPILATION -> CBO -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_CONNECT_BY_CB_WHR_ONLY          ALL -> COMPILATION -> TRANSFORMATION
CONNECT_BY_CB_WHR_ONLY             ALL -> COMPILATION -> TRANSFORMATION
INLINE                             ALL -> COMPILATION -> TRANSFORMATION
NO_TABLE_LOOKUP_BY_NL              ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
TABLE_LOOKUP_BY_NL                 ALL -> COMPILATION -> TRANSFORMATION -> CBQT -> STAR_TRANS -> TABLE_LOOKUP_BY_NL
NO_NLJ_BATCHING                    ALL -> EXECUTION
NLJ_BATCHING                       ALL -> EXECUTION
NO_NLJ_PREFETCH                    ALL -> EXECUTION
NLJ_PREFETCH                       ALL -> EXECUTION

Plenty of interesting stuff here – the new hint TABLE_LOOKUP_BY_NL that has showed up recently seems to have to do with star transformations for example (I just learned this myself from this output).

Interestingly the NLJ_BATCHING and NLJ_PREFETCH hints are considered as execution phase hints apparently (that was my term, I’m thinking about hints (also) affecting a decision in the execution phase, not just during optimization). For example, normally the NLJ prefetch feature can be dynamically turned on & off during the query execution, I guess with a hint this feature would be always enabled (I’m not sure about this here, just trying to reason why a hint is shown to be related to “execution” phase).

If optimizer feature terms like CBQT and CVM do not immediately ring a bell, you can use the V$SQL_FEATURE view (or my sqlfh.sql script) to list some more info about what these SQL feature name abbreviations mean and where in the hierarchy does this particular feature stand.

The script below doesn’t accept any parameters, prints out the entire SQL feature hierarchy (except the temporary bugfix features you can see from V$SYSTEM_FIX_CONTROL):

SQL> @sqlfh

SQL_FEATURE                                             DESCRIPTION
----------------------------------- ----------------------------------------------------------------
ALL                                 A Universal Feature
  COMPILATION                       SQL COMPILATION
    CBO                             SQL Cost Based Optimization
      ACCESS_PATH                   Query access path
        AND_EQUAL                   Index and-equal access path
        BITMAP_TREE                 Bitmap tree access path
        FULL                        Full table scan
        INDEX                       Index
        INDEX_ASC                   Index (ascending)
        INDEX_COMBINE               Combine index for bitmap access
        INDEX_DESC                  Use index (descending)
        INDEX_FFS                   Index fast full scan
        INDEX_JOIN                  Index join
        INDEX_RS_ASC                Index range scan
        INDEX_RS_DESC               Index range scan descending
        INDEX_SS                    Index skip scan
        INDEX_SS_ASC                Index skip scan ascending
        INDEX_SS_DESC               Index skip scan descending
        SORT_ELIM                   Sort Elimination Via Index
      CBQT                          Cost Based Query Transformation
        CVM                         Complex View Merging
        DIST_PLCMT                  Distinct Placement
        JOINFAC                     Join Factorization
        JPPD                        Join Predicate Push Down
        PLACE_GROUP_BY              Group-By Placement
        PULL_PRED                   pull predicates
        STAR_TRANS                  Star Transformation
          TABLE_LOOKUP_BY_NL        Table Lookup By Nested Loop
        TABLE_EXPANSION             Table Expansion
        UNNEST                      unnest query block
      CURSOR_SHARING                Cursor sharing
      DML                           DML
      JOIN_METHOD                   Join methods
        USE_HASH                    Hash join
        USE_MERGE                   Sort-merge join
        USE_MERGE_CARTESIAN         Merge join cartesian
        USE_NL                      Nested-loop join
        USE_NL_WITH_INDEX           Nested-loop index join
      JOIN_ORDER                    Join order
      OPT_MODE                      Optimizer mode
        ALL_ROWS                    All rows (optimizer mode)
        CHOOSE                      Choose (optimizer mode)
        FIRST_ROWS                  First rows (optimizer mode)
      OR_EXPAND                     OR expansion
      OUTLINE                       Outlines
      PARTITION                     Partition
      PQ                            Parallel Query
        PARALLEL                    Parallel table
        PQ_DISTRIBUTE               PQ Distribution method
        PQ_MAP                      PQ slave mapper
        PX_JOIN_FILTER              Bloom filtering for joins
      STAR_TRANS                    Star Transformation
        TABLE_LOOKUP_BY_NL          Table Lookup By Nested Loop
      STATS                         Optimizer statistics
        CARDINALITY                 Cardinality computation
        COLUMN_STATS                Basic column statistics
        CPU_COSTING                 CPU costing
        DBMS_STATS                  Statistics gathered by DBMS_STATS
        DYNAMIC_SAMPLING            Dynamic sampling
        DYNAMIC_SAMPLING_EST_CDN    Estimate CDN using dynamic sampling
        GATHER_PLAN_STATISTICS      Gather plan statistics
        INDEX_STATS                 Basic index statistics
        OPT_ESTIMATE                Optimizer estimates
        TABLE_STATS                 Basic table statistics
    QUERY_REWRITE                   query rewrite with materialized views
    RBO                             SQL Rule Based Optimization
    SQL_CODE_GENERATOR              SQL Code Generator
    SQL_PLAN_MANAGEMENT             SQL Plan Management
    TRANSFORMATION                  Query Transformation
      CBQT                          Cost Based Query Transformation
        CVM                         Complex View Merging
        DIST_PLCMT                  Distinct Placement
        JOINFAC                     Join Factorization
        JPPD                        Join Predicate Push Down
        PLACE_GROUP_BY              Group-By Placement
        PULL_PRED                   pull predicates
        STAR_TRANS                  Star Transformation
          TABLE_LOOKUP_BY_NL        Table Lookup By Nested Loop
        TABLE_EXPANSION             Table Expansion
        UNNEST                      unnest query block
      HEURISTIC                     Heuristic Query Transformation
        CNT                         Count(col) to count(*)
        COALESCE_SQ                 coalesce subqueries
        CSE                         Common Sub-Expression Elimination
        CVM                         Complex View Merging
        FILTER_PUSH_PRED            Push filter predicates
        FULL_OUTER_JOIN_TO_OUTER    Join Conversion
        JPPD                        Join Predicate Push Down
        OBYE                        Order-by Elimination
        OLD_PUSH_PRED               Old push predicate algorithm (pre-10.1.0.3)
        OUTER_JOIN_TO_ANTI          Join Conversion
        OUTER_JOIN_TO_INNER         Join Conversion
        PRED_MOVE_AROUND            Predicate move around
        SET_TO_JOIN                 Transform set operations to joins
        SVM                         Simple View Merging
        TABLE_ELIM                  Table Elimination
        UNNEST                      unnest query block
        USE_CONCAT                  Or-optimization
    XML_REWRITE                     XML Rewrite
      CHECK_ACL_REWRITE             Check ACL Rewrite
      COST_XML_QUERY_REWRITE        Cost Based XML Query Rewrite
      XMLINDEX_REWRITE              XMLIndex Rewrite
  EXECUTION                         SQL EXECUTION

I highlighted the CVM and CBQT lines above…

Just for reference (and if you’re too lazy to run these scripts yourself), I’ve pasted the full output of the hint feature hierarchy script too (executed in my 11.2.0.3 DB):

Listing Exadata storage cells and their configuration info from V$CELL_CONFIG

Thu, 21 Mar 2013 22:18:47 +0000

If you attended my Exadata hacking session today, you saw me using the cellver.sql script which lists some basic configuration info about the currently connected storage cells: SQL> @exadata/cellver Show Exadata cell versions from V$CELL_CONFIG.... CELLNAME CELLSRV_VERSION FLASH_CACHE_MODE CPU_COUNT UPTIME KERNEL_VERSION MAKE_MODEL -------------------- -------------------- -------------------- ---------- -------------------- ------------------------------ -------------------------------------------------- 192.168.12.10 11.2.3.2.1 WriteBack 24 8 days, 2:07 2.6.32-400.11.1.el5uek Oracle Corporation SUN FIRE X4270 M2 SERVER SAS 192.168.12.11 11.2.3.2.1 WriteBack 24 8 days, 2:06 2.

Alter session force parallel query doesn’t really force anything

Wed, 20 Mar 2013 16:04:55 +0000

Jonathan Lewis has already written about this behavior from the angle of PARALLEL hints. I’m writing a similar article just because the word FORCE in the ALTER SESSION FORCE PARALLEL QUERY syntax. Force should mean that some behavior would always happen (when possible), right? Let’s test: SQL> CREATE TABLE t AS SELECT * FROM dba_objects; Table created. SQL> CREATE INDEX i ON t(owner); Index created. SQL> @gts t Gather Table Statistics for table t.

asqlmon.sql: SQL Monitoring-like execution plan line level drilldown into SQL response time

Sun, 17 Mar 2013 14:23:52 +0000

I don’t have much time for a thorough blog post, so I’ll just paste in an example output of my asqlmon.sql script, which uses ASH sql_plan_line columns for displaying where inside your execution plan response time has been spent. Why not just use Oracle’s own SQL Monitoring reports? Well, SQL monitoring is meant for “long running” queries, which are not executed very frequently. In other words, you can’t use SQL Monitoring for drilling down into your frequently executed OLTP-style SQL.

ExaSnapper 0.7 beta download and the hacking session videos

Fri, 22 Feb 2013 19:38:28 +0000

Thank you for attending the Exadata Snapper (ExaSnapper) hacking session!

I have split the recording of this session into 3 pieces and uploaded to enkitec.tv. The ExaSnapper beta that I demoed is also available now in my blog. See the links below.

For quick reference, here’s the syntax of running ExaSnapper – there are two modes, one is the before/after capture (think Tom Kyte’s runstats, but for exadata metrics) and the other is more like a DBA-monitoring mode, where you can just measure a few seconds worth of a long-running query runtime and get the IO and efficiency figures from there.

Here’s an excerpt from the install script documentation section:

-- Usage:       Take a snapshot of a running session (use QC SID if PX):
--
--                a) Monitor a running query - "DBA mode" 
--
--                   SELECT * FROM TABLE(exasnap.display_sid(, [snap_seconds], [detail_level]));
--
--                   The SID argument can be just a number (SID in local instance) or a remote SID with
--                   @instance after it (like '123@4')
--
--                   SELECT * FROM TABLE(exasnap.display_sid(123));
--                   SELECT * FROM TABLE(exasnap.display_sid('123@4', p_detail=>'%');
--
--                b) Take Before & After snapshots of a query execution - "Developer Mode"
--
--                   1) SELECT exasnap.begin_snap(123) FROM dual;
--                        or
--                      EXEC :begin_snap_id := exasnap.begin_snap(123);
-- 
--                   2) Run your query, wait until it finishes (or CTRL+C)
--
--                   3) SELECT exasnap.end_snap(123) FROM dual;
--                        or
--                      EXEC :end_snap_id := exasnap.end_snap(123);
--
--                   4) SELECT * FROM TABLE(exasnap.display_snap(:begin_snap_id, :end_snap_id, '%'));
--

One slide illustrating the idea of ExaSnapper:

Peeking into Linux kernel-land using /proc filesystem for quick’n’dirty troubleshooting

Thu, 21 Feb 2013 14:46:48 +0000

This blog entry is about modern Linuxes. In other words RHEL6 equivalents with 2.6.3x kernels and not the ancient RHEL5 with 2.6.18 kernel (wtf?!), which is the most common in enterprises unfortunately. And no, I’m not going to use kernel debuggers or SystemTap scripts here, just plain old “cat /proc/PID/xyz” commands against some useful /proc filesystem entries.

Troubleshooting a “slow” process

Here’s one systematic troubleshooting example I reproduced in my laptop. A DBA was wondering why their find command had been running “much slower”, without returning any results for a while. Knowing the environment, we had a hunch, but I got asked about what would be the systematic approach for troubleshooting this – already ongoing – problem right now.

Luckily the system was running OEL6, so had a pretty new kernel. Actually the 2.6.39 UEK2.

So, let’s do some troubleshooting. First let’s see whether that find process is still alive:

[root@oel6 ~]# ps -ef | grep find
root     27288 27245  4 11:57 pts/0    00:00:01 find . -type f
root     27334 27315  0 11:57 pts/1    00:00:00 grep find

Yep it’s there – PID 27288 (I’ll use that pid throughout the troubleshooting example).

Let’s start from the basics and take a quick look what’s the bottleneck for this process – if it’s not blocked by anything (for example reading everything it needs from cache) it should be 100% on CPU. If it’s bottlenecked by some IO or contention issues, the CPU usage should be lower – or completely 0%.

[root@oel6 ~]# top -cbp 27288
top - 11:58:15 up 7 days,  3:38,  2 users,  load average: 1.21, 0.65, 0.47
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2026460k total,  1935780k used,    90680k free,    64416k buffers
Swap:  4128764k total,   251004k used,  3877760k free,   662280k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
27288 root      20   0  109m 1160  844 D  0.0  0.1   0:01.11 find . -type f

Top tells me this process is either 0% on CPU or very close to zero percent (so it gets rounded to 0% in the output). There’s an important difference though, as a process that is completely stuck, not having a chance of getting onto CPU at all vs. a process which is getting out of its wait state every now and then (for example some polling operation times out every now and then and thee process chooses to go back to sleep). So, top on Linux is not a good enough tool to show that difference for sure – but at least we know that this process is not burning serious amounts of CPU time.

Let’s use something else then. Normally when a process seems to be stuck like that (0% CPU usually means that the process is stuck in some blocking system call – which causes the kernel to put the process to sleep) I run strace on that process to trace in which system call the process is currently stuck. Also if the process is actually not completely stuck, but returns from a system call and wakes up briefly every now and then, it would show up in strace (as the blocking system call would complete and be entered again a little later):

[root@oel6 ~]# strace -cp 27288
Process 27288 attached - interrupt to quit

^C
^Z
[1]+  Stopped                 strace -cp 27288

[root@oel6 ~]# kill -9 %%
[1]+  Stopped                 strace -cp 27288
[root@oel6 ~]# 
[1]+  Killed                  strace -cp 27288

Oops, the strace command itself got hung too! It didn’t print any output for a long time and didn’t respond to CTRL+C, so I had to put it into background with CTRL+Z and kill it from there. So much for easy diagnosis.

Even more Snapper – v4.03 now works in SQL Developer too!

Mon, 18 Feb 2013 23:21:54 +0000

If you already downloaded snapper v4, then better re-download it again as the v4.03 also runs in SQL Developer! You might also want to check how to enable the DBMS_OUTPUT display in SQL Developer post by “That” Jeff Smith :-) Update: Niall Litchfield pointed out that the “set serverout on” command was ignored on SQL Developer because I was using the short syntax (serverout instead of serveroutput). Once I changed this, the SQL Developer doesn’t print the warning anymore and there’s no need to explicitly open up the separate DBMS_OUTPUT.

Manual “before” and “after” snapshot support in Snapper v4

Mon, 18 Feb 2013 22:10:18 +0000

Snapper used to require access to DBMS_LOCK, so it could sleep for X seconds between the “before” and “after” performance data snapshots. Now it is possible to get away without using DBMS_LOCK. Instead you will run Snapper twice, once for taking the “before” snapshot, then run your workload and then run Snapper again for taking the “after” snapshot and print the output. So, the usual way of running snapper is this:

Snapper v4.02 and the Snapper launch party video

Mon, 18 Feb 2013 21:07:00 +0000

I have fixed most of the bugs that showed up during the Snapper launch party session and uploaded the new version (v4) of Snapper here: https://github.com/tanelpoder/tpt-oracle/blob/master/snapper.sql I have uploaded some of my videos to https://www.youtube.com/tanelpoder: Snapper & Ashtop Overview Snapper v4 Launch Party video (old) I have not updated the snapper documentation yet, but here are the main improvements: RAC support (query remote instance’s sessions performance data via GV$) Manually specified “before and after” snapshot support.

Drilling Deep Into Exadata Performance with ASH, SQL Monitoring and Exadata Snapper – slides and a hacking session!

Thu, 14 Feb 2013 23:04:04 +0000

The promised hacking session about Exadata performance troubleshooting will happen on Thursday 21st February 9am-11am PST (2 hours). I will show a number of examples and demos where the Exadata Snapper shows what was going on in storage cells when running Smart Scans or just doing IO. Register here (free stuff!): https://www1.gotomeeting.com/register/270528161 Slides: /files/Tanel_Poder_Drilling_Deep_Into_Exadata_Performance.pdf I will post the scripts on the day of this session. Note that you should read through the slides before the hacking session – as I’m not going to spend much time on presenting slides, as the name says – it’s a hacking session, so 95% demos.

Troubleshooting high CPU usage with poor-man’s stack profiler – in a one-liner!

Thu, 14 Feb 2013 21:12:04 +0000

Here’s an example of a quick’n’dirty way of profiling stack traces on your command line. This is an example from Solaris (but the script should work on Linux too plus other Unixes with minor modifications).

I created a problem case below, based on a case I once troubleshooted at a client site. Note that they had set optimizer_mode = FIRST_ROWS in their database and the optimized came up with a very inefficient execution plan for the select from DBA_LOCK_INTERNAL view below:

SQL> ALTER SESSION SET optimizer_mode = FIRST_ROWS;

Session altered.

SQL> SET TIMING ON

SQL> SELECT * FROM dba_lock_internal;
...
... the output data stripped ...
...

927 rows selected.

Elapsed: 00:23:27.14

It took over 23 minutes to return 927 rows from DBA_LOCK_INTERNAL!

I ran Snapper to see where the time is spent then:

SQL> @snapper4 all 5 1 222
Sampling SID 222 with interval 5 seconds, taking 1 snapshots...

-- Session Snapper v4.00 BETA - by Tanel Poder (  ) - Enjoy the Most Advanced Oracle Troubleshooting Script on the Planet! :)

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    SID, USERNAME  , TYPE, STATISTIC                                                 ,         DELTA, HDELTA/SEC,    %TIME, GRAPH       , NUM_WAITS,  WAITS/SEC,   AVERAGES
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    222, SYS       , STAT, non-idle wait count                                       ,             1,        .19,         ,             ,          ,           ,          ~ per execution
    222, SYS       , TIME, DB CPU                                                    ,       6028084,      1.17s,   117.2%, [@@@@@@@@@@],          ,           ,
    222, SYS       , TIME, sql execute elapsed time                                  ,       6032677,      1.17s,   117.3%, [##########],          ,           ,
    222, SYS       , TIME, DB time                                                   ,       6032677,      1.17s,   117.3%, [##########],          ,           ,          ~ unaccounted time
    222, SYS       , WAIT, library cache: mutex X                                    ,             6,     1.17us,      .0%, [          ],         1,        .19,        6us average wait

--  End of Stats snap 1, end=2013-02-14 21:30:45, seconds=5.1

---------------------------------------------------------------------------------------------
Active% | SQL_ID          | SQL_CHILD | EVENT                               | WAIT_CLASS
---------------------------------------------------------------------------------------------
   100% | c884zcqpv9y5h   | 0         | ON CPU                              | ON CPU

--  End of ASH snap 1, end=2013-02-14 21:30:45, seconds=5, samples_taken=50

The query is apparently 100% on CPU with no significant waits. Normally I would expect some other metrics to pop up in snapper output in such high-CPU cases, like thousands of session logical reads per second, sorts (rows) showing millions of rows sorted per second or parse count (hard) number in hundreds or thousands per second. These are the “usual suspects”.

But this time none of these additional metrics were incremented by the session. So it’s time to systematically drill down by other means.

Session Snapper v4 – The World’s Most Advanced Oracle Troubleshooting Script!

Sun, 10 Feb 2013 17:33:18 +0000

Snapper V4 Webinar Snapper V4 is out! The major new features include: RAC support – ability to query stats from remote instances Manual Before and After snapshot support – no need to use DBMS_LOCK sleeps anymore Show useful averages and ratios *in addition to* raw metrics for faster troubleshooting And more! :) Snapper is still a free-to-use tool and it still does NOT require any object creation nor changes in your databases for use.

Sqlplus is my second home: Part 7 – Downloading files via sqlplus :-)

Thu, 24 Jan 2013 21:17:22 +0000

Some years ago I wrote about how sqlplus allows you to run sqlplus scripts directly from HTTP and FTP locations instead of the local filesystem. By the way, I didn’t even notice – my blog is over 5 years old already! :) I a recent email thread Marco Gralike just showed the simplest way I to open a HTTP URL and download + list its contents in a CLOB datatype. It’s the HTTPURITYPE and its getCLOB (and getBLOB) methods.

Japanese translation of some of my blog articles

Thu, 17 Jan 2013 22:49:35 +0000

Ryota Watabe has translated some of my articles into Japanese and intends to translate more in the future (thanks for that! :) I have added a little “translations” section to the left sidebar of the blog (scroll down). Or just go to this link: http://www.csus4.net/d/japanese_translations_of_tanel_poder_s_posts_and_articles/ You might also want to follow Watabe-san on Twitter to get notified of any new Japanese translations that show up.

What the heck is the INTERNAL_FUNCTION in execution plan predicate section?

Thu, 17 Jan 2013 00:40:11 +0000

Sometimes you see something like this in an execution plan: -------------------------------------------------------------------------- | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | -------------------------------------------------------------------------- | 0 | SELECT STATEMENT | | | | 2 (100)| | |* 1 | TABLE ACCESS FULL| T | 1 | 22 | 2 (0)| 00:00:01 | -------------------------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("B"=INTERNAL_FUNCTION("A")) There’s quite a little information available about what the INTERNAL_FUNCTION really is and why does it show up, thus this blog entry.

A tip for lazy Oracle users – type less with ANSI DATE and TIMESTAMP SQL syntax

Sat, 29 Dec 2012 19:00:20 +0000

I am lazy, therefore if I can type less, I will type less. Often people are surprised to see that Oracle supports the ANSI DATE and TIMESTAMP syntax in the SQL code, which allows me to shorten the lengthy TO_DATE( …. , ‘YYYY-MM-DD HH24:MI:SS’) syntax a bit. You can just type this if you want to compare some field to a date (day precision): SQL> SELECT COUNT(*) FROM dba_objects WHERE created > DATE'2012-12-01'; COUNT(*) 0 SQL> SELECT COUNT(*) FROM dba_objects WHERE created > DATE'2012-01-01’;

Snapper v3.61 released – and more work is in progress!

Wed, 19 Dec 2012 01:25:00 +0000

Here’s the latest version of my Oracle session level performance Snapper tool: https://github.com/tanelpoder/tpt-oracle/blob/master/snapper.sql I’m going to add more stuff to Snapper in coming days, but thought to release something for testing already :) There are some cosmetic changes, like printing empty lines and header lines for better readability (still controllable by the pagesize parameter, look into the scripts). I’ve made some changes in the visual “graph” column just for easier readability when glancing at a snapper report: the @-character is used for CPU time (@ looks like a letter C a bit, doesn’t it :)

Select statement generating redo – and lost write detection

Wed, 31 Oct 2012 12:08:46 +0000

Gwen Shapira has written a nice summary of a problem case where the classic wait interface based troubleshooting method is not always enough for troubleshooting low-level issues. The top SQL + top wait approach should usually be used as the starting point of troubleshooting a session, workload etc, but often the troubleshooting does not stop there. So, when the wait interface and related tools don’t explain the problem well enough, then you either start guessing from there or dig deeper into performance data.

Sqlplus is my second home: Part 6 – Colorizing sqlplus and terminal output

Wed, 19 Sep 2012 22:39:20 +0000

Jagjeet Singh mentioned that he has built a V2 of his cool SQL Dashboard sqlplus-based monitoring script, which now also shows a nice colored ASH graph output. Yes, right inside sqlplus! This reminded me that I demoed color output in sqlplus when I delivered the Hotsos Symposium training day a few years ago, when introducing the MOATS tool the first time. I never got to colorize it as I had hoped to though… as MOATS was more of a fun project I worked on with Adrian Billington just to prove that it is possible to build a monitoring tool purely based on SQL and PL/SQL with a TOP-like self-refreshing screen output.

Optimizer statistics-driven direct path read decision for full table scans (_direct_read_decision_statistics_driven)

Tue, 04 Sep 2012 00:30:19 +0000

Hello all fellow Oracle geeks and technology enthusiasts! Long time no see ;-) In the hacking session about Oracle full table scans and direct path reads I explained how the direct path read decision is not done by the optimizer, but instead during every execution, separately for every single segment (partition) scanned in the query. I also explained how the _small_table_threshold parameter and the X$KCBOQH.NUM_BUF(which keeps track of how many buffers of any segment are currently cached) are used for determining whether to scan using direct path reads or not.

The limitations of CURSOR_SHARING = FORCE and FORCE_MATCHING_SIGNATURE for SQL plan stability

Fri, 03 Aug 2012 04:05:39 +0000

It’s a well known fact that the cursor_sharing parameter can be set to FORCE for making Oracle replace any literals with system-generated bind variable placeholders and then calculating the SQL hash value for looking up an existing cursor in library cache. This should reduce the amount of hard parsing and shared pool garbage. Also, the same mechanism (of replacing literal values with bind variable placeholders before calculating the SQL hash value for library cache lookup) can be used for enforcing a SQL profile for SQL statements which differ by literal values in them.

MOATS-like sqlplus “top” utility for RAC

Sun, 13 May 2012 11:42:11 +0000

So, you think MOATS was cool?! Check this out by Jagjeet Singh :)

Oradebug hanganalyze with a prelim connection and “ERROR: Can not perform hang analysis dump without a process state object and a session state object.”

Tue, 08 May 2012 19:45:22 +0000

Back in the (really) old days, systemstate dumps had to be used for diagnosing hangs and finding blockers of hung databases. Basically you just identified which resource your waiting sessions were waiting for and then scanned through the rest of the system state dump to see which session held that particular resource (note that over time the instrumentation and systemstate dumps have evolved to resolve and dump the blocker information right at the waiting session section in the dump).

Advanced Oracle Troubleshooting Guide – Part 10: Index unique scan doing multiblock reads?!

Wed, 02 May 2012 19:42:56 +0000

When you troubleshoot an Oracle (performance) problem you usually want to find out what are the problem sessions doing (wrong) and then why are they doing it. The “what” question is usually answered using some wait interface based method – like ASH or SQL*Trace, which both add plenty of extra details to the plain wait event data. My normal session troubleshooting data collection sequence consists of only three steps: Interpret session-level top wait and top SQL data (ASH, SQL*Trace or custom tools sampling session-level V$views) Interpret session-level performance counter data (V$SESSTAT) Sample the problem process stack and list the top stack branches where execution spent the most time Only looking into #1 is enough in 90% of problem cases.

Where is LOB data stored?

Sun, 22 Apr 2012 15:17:31 +0000

There was a question in Oracle-L about where is the LOB data actually stored (in the row or the LOB segments) and what are the exact conditions when a switch from one to another may happen. The documentation isn’t fully clear about this and the “4000 bytes” number may mislead people to think that you can store 4000 bytes of your data in a LOB item before it must move out-of-line.

Exadata Smart Scan predicate offloading and sequence.NEXTVAL

Fri, 13 Apr 2012 15:49:19 +0000

There was a question in the twitter-sphere about whether using sequences (sequence.NEXTVAL) in your select query’s projection list would somehow disable smart scans happening? The answer is no, sequence use with smart scans works just fine. The smart scan offloading applies to data retrieval row sources (and filtering) only and nothing else. So, what you have in the query’s projection list (the sequence use for example), does not directly affect the smart scan decision.

Create a database link with the new host:port/service syntax

Mon, 05 Mar 2012 15:57:11 +0000

I just noticed that (finally) in 11.2 this syntax is supported: SQL> CREATE DATABASE LINK demo_x2 2 CONNECT TO tanel IDENTIFIED BY password 3 USING 'exadb03:1521/DEMO'; Database link created. This just makes life a bit easier as there’s no need to use the long TNS format entry (or a tnsnames.ora/LDAP alias). It might work in 11.1 too (haven’t tested) but it didn’t work on 10.2.0.4 … Update: This feature works for dblinks in 10.

Oracle Core: Essential Internals for DBAs and Developers book by Jonathan Lewis

Thu, 22 Dec 2011 23:48:09 +0000

In case you didn’t know, Jonathan Lewis’es new Oracle Core: Essential Internals for DBAs and Developers book is out (for a few weeks already). I was the technical reviewer for that book and I can say it’s awesome! It will likely be the best Oracle internals book out there for the coming 10 years, just like Steve Adams’es Oracle Internal Services book was in the last decade :) Jonathan does a very good job explaining complex things in a simple enough way – and the book is not just dry listing of how things work inside Oracle database, but also why they work like they do and what are the benefits, limitations and side effects of the behavior.

Profiling trace files with preprocessor external tables in 11g and some parallel execution hacking

Mon, 14 Nov 2011 21:33:00 +0000

If you work with SQL Trace files (and profile them) then you should check out the awesome novel use of the “external table preprocessor” feature explained by Adrian Billington here: http://www.oracle-developer.net/display.php?id=516 Ironically just a day after writing my “Evil things” article, I noticed a note in MOS about how to enable an event 10384 at level 16384 to get a parallel plan to be executed in serial: How to force that a Parallel Query runs in serial with the Parallel Execution Plan [ID 1114405.

Evil things are happening in Oracle

Sun, 13 Nov 2011 16:05:09 +0000

Relax, I’m talking about the Oracle Database kernel here, not the corporation ;-) Here’s a couple of more reasons why not to play around with undocumented debug events unless you’re really sure why and how would they help to solve your specific problem (and you’ve gotten a blessing in some form from Oracle support too): $ oerr ora 10665 10665, 00000, "Inject Evil Literals" // *Cause: Event 10665 is set to some number > 0, causing 1/(value-1) of all // literals to be replaced by 2000 letter 'A's.

What the heck is the SQL Execution ID – SQL_EXEC_ID?

Mon, 24 Oct 2011 20:55:29 +0000

Ok, I think it’s time to write another blog entry. I’ve been traveling and dealing with jetlag from 10-hour time difference, then traveling some more, spoken at conferences, drank beer, had fun, then traveled some more, trained customers, hacked some Exadatas and now I’m back home. Anyway, do you know what is the SQL_EXEC_ID in V$SESSION and ASH views? Oh yeah, it’s the “SQL Execution ID” just like the documentation says … all clear.

V8 Bundled Exec call – and Oracle Program Interface (OPI) calls

Tue, 23 Aug 2011 13:13:23 +0000

So, what he hell is that V8 Bundled Exec call which shows up in various Oracle 11g monitoring reports?! It’s yet another piece of instrumentation which can be useful for diagnosing non-trivial performance problems. Oracle ASH has allowed us to measure what is the top wait event or top SQLID for a long time, but now it’s also possible to take a step back and see what type of operation the database session is servicing.

Full scans, direct path reads and ORA-8103 error hacking session video

Thu, 11 Aug 2011 17:35:46 +0000

I have uploaded the latest hacking session video. I have edited it a little, I cut out the part where I spilled an entire Red Bull onto my desk, with some onto my laptop (some keys are still sticky)! Note that this is an old video from year 2011 and it’s recording resolution is not too good. But you’ll still find useful stuff from in there despite the blurriness :)

Are you getting the most out of your Exadata performance? Part 1

Mon, 08 Aug 2011 14:54:42 +0000

In almost all of the Exadata migration projects I’ve been part of, the client sees immediate speedup & performance increase when testing their workload on Exadata (of course, we’ve made sure that we do plan & execute the tasks right). However, my performance geek’s nature usually doesn’t allow to stop there and leave the client with just 2x or 3x performance increase. For data warehousing and reporting workloads, Exadata can do much better than just 2-3x performance increase!

What is the purpose of segment level checkpoint before DROP/TRUNCATE of a table?

Wed, 06 Jul 2011 08:52:10 +0000

There was a very good question asked in Oracle-L list today, which has bothered me too in past. The question was: What is the purpose of a segment level checkpoint before DROP/TRUNCATE of a table? In other words, why do we have to wait for the enq: RO - fast object reuse wait event (and in 11.2 the enq: CR - block range reuse ckpt wait) when dropping and truncating segments?

Another cache buffers chains latch contention troubleshooting example using LatchProf

Thu, 30 Jun 2011 21:53:34 +0000

One of my blog readers recently dropped me an email noting that he had noticed some cache buffers chains latch contention recently and successfully troubleshooted it with LatchProf. I asked if he’d like to blog about it and here’s the article: http://web.archive.org/web/20111113062613/http://orapsdba.wordpress.com/2011/06/21/another-latchcache-buffer-chains-troubleshooting Cache buffer chains latch contention typically shows up when some execution plans go bad, switching to nested loops or filter loops and revisiting the same table (or index) blocks very frequently…

Tech Reviewer, Tech Reviewer! ;-)

Wed, 29 Jun 2011 10:32:49 +0000

I just noticed that Jonathan Lewis has announced that he’s writing a new Oracle (fundamental) internals book, due to be out in November. So, I’m happy to add to Jonathan’s announcement, that I’m the tech reviewer of that book! After all the hard work on the Exadata book, I didn’t want to hear about working on any book again (even if it’s just tech reviewing work), but as this is Jonathan’s book, about exactly these topics I love and focus on, I had no choice but to make an exception and become a reviewer ;-)

IOUG Select Journal Editor’s Choice Award 2011

Wed, 29 Jun 2011 10:23:18 +0000

In May I received the IOUG Select Journal Editor’s Choice Award for my Systematic Oracle Latch Contention Troubleshooting article where I introduced my LatchProfX tool for advanced drilldown into complex latch contention problems (thanks IOUG and John Kanagaraj!). As the relevant IOUG webpage hasn’t been updated yet, I thought to delay this announcement until the update was done – but I just found an official enough announcement (press release) by accident from Reuters site:

Knowing what you want to achieve before thinking of how to achieve it – a query optimization example

Tue, 28 Jun 2011 23:48:16 +0000

Today I received a question which was a good example of systematic problem solving approach. It was about getting a long-running query to run faster. It took a long time as the correlated subquery in the query was not unnested, was re-visited many times, causing the whole subquery subtree in the plan to be executed again and again). The main part of the question was this: Is there a way to avoid “NOT IN” conversion to “NOT EXISTS” by optimizer … My sub query, I would like it to be fully instantiated as view and then execute it as a Hash Anti join.

Running SELECT … INTO :bind_variable from SQL

Sun, 10 Apr 2011 15:46:09 +0000

I just wasted a few minutes troubleshooting one of my scripts – and realized that while SELECT … INTO :bind_variable does not error out when executed as top-level SQL, it doesn’t seem to populate the resulting bind variable: SQL> VAR blah VARCHAR2(10) SQL> SELECT dummy INTO :blah FROM dual; D - X SQL> print blah BLAH -------------------------------- See, the bind variable is empty… Now, let’s run the same statement via PL/SQL engine:

Latch contention troubleshooting case study and Flashback Database performance issues with LOBs

Mon, 04 Apr 2011 13:26:27 +0000

Steve Bamber has written up a case study of library cache latch contention troubleshooting of an Apex application with LatchProf. I’m happy that others also see the value and have had success with my new LatchProf based latch contention troubleshooting approach which takes into account both sides of the contention story (latch waiters and latch holders/blockers) as opposed to the guesswork used previously (hey if it’s shared pool latch contention – is must be about bad SQL not using bind variables ….

MOATS: The Mother of All Tuning Scripts!

Tue, 29 Mar 2011 23:23:37 +0000

Update: Based on the original MOATS built by Adrian Billington and myself, other folks have built cool variations of MOATS (either based on our code or written completely new ones): MOATS (original) by Adrian Billington & Tanel: https://github.com/oracle-developer/moats MOATS 2.0 (RAC-aware) by Sidney Chen: https://github.com/dbsid/moats_rac SQL Dashboard by Jagjeet Singh: https://jagjeet.wordpress.com/2013/12/20/sql-dashboard-v2/ Both are RAC-aware, use terminal coloring techniques like my (fish.sql :) and some Exadata metrics. A couple of screenshots from their tools:

An index of my TPT scripts

Thu, 24 Mar 2011 14:10:05 +0000

A lot of people have asked me whether there’s some sort of index or “table of contents” of my TPT scripts (there’s over 1000 in my TPT script repo right now). I have planned to create such index for years, but never got to it. I probably never will :) So a good way to extract the descriptions of some scripts is to just grep for Purpose: in the scripts directory:

LOBREAD SQL Trace entry in Oracle 11.2 (and tracing OPI calls with event 10051)

Sun, 20 Mar 2011 14:52:49 +0000

A few days ago I looked into a SQL Tracefile of some LOB access code and saw a LOBREAD entry there. This is a really welcome improvement (or should I say, bugfix of a lacking feature) for understanding resource consumption by LOB access OPI calls. Check the bottom of the output below: *** 2011-03-17 14:34:37.242 WAIT #47112801352808: nam='SQL*Net message from client' ela= 189021 driver id=1413697536 #bytes=1 p3=0 obj#=99584 tim=1300390477242725 WAIT #0: nam='gc cr multi block request' ela= 309 file#=10 block#=20447903 class#=1 obj#=99585 tim=1300390477243368 WAIT #0: nam='cell multiblock physical read' ela= 283 cellhash#=379339958 diskhash#=787888372 bytes=32768 obj#=99585 tim=1300390477243790 WAIT #0: nam='SQL*Net message to client' ela= 2 driver id=1413697536 #bytes=1 p3=0 obj#=99585 tim=1300390477243865 [.

Implicit datatype conversion in the parsing phase – something new I learned today!

Fri, 18 Mar 2011 12:25:33 +0000

Wow, I wasn’t aware that Oracle can also do an implicit datatype conversion for literal strings during parsing phase! SQL> @desc t Name Null? Type ------------------------------- -------- ---------------------------- 1 A NUMBER(38) SQL> select * from t where a = '1' || RPAD('0',5,'0'); no rows selected SQL> @x Display execution plan for last statement for this session from library cache... PLAN_TABLE_OUTPUT ------------------------------------------------------------------------------------------------ SQL_ID d7r6md8wfu74d, child number 0 ------------------------------------- select * from t where a = '1' || RPAD('0',5,'0') Plan hash value: 1601196873 -------------------------------------------------------- | Id | Operation | Name | E-Rows | Cost (%CPU)| -------------------------------------------------------- | 0 | SELECT STATEMENT | | | 2 (100)| |* 1 | TABLE ACCESS FULL| T | 1 | 2 (0)| -------------------------------------------------------- Predicate Information (identified by operation id): --------------------------------------------------- 1 - filter("

ORA-4031 errors, contention, cursor management issues and shared pool fragmentation – free secret seminar!

Wed, 16 Mar 2011 12:21:50 +0000

Note that the live event is over by now, but you can see the recording from Enkitec.tv On Tuesday 22nd March I’ll hold two (yes two) Secret Oracle Hacking Sessions – about ORA-04031: unable to allocate x bytes of shared memory errors, cursor management issues and other shared pool related problems (like fragmentation). This event is free for all! You’ll just need to be fast enough to register, both events have 100 attendee limit (due to my GotoWebinar accont limitations).

Exadata CAN do smart scans on bitmap indexes

Tue, 15 Mar 2011 22:27:43 +0000

As I’m finishing up a performance chapter for the Exadata book (a lot of work!), I thought to take a quick break and write a blog entry. This is not really worth putting into my Oracle Exadata Performance series (which so far has only 1 article in it anyway) .. so this is a little stand-alone article … Everybody knows that the Exadata smart scan can be used when scanning tables (and table partitions).

Oracle Exadata Performance series – Part 1: Should I use Hugepages on Linux Database Nodes?

Sun, 13 Mar 2011 18:54:42 +0000

There was a question in LinkedIn forum about whether Linux Hugepages should be used in Oracle Exadata Database layer, as they aren’t enabled by default during ACS install. I’m putting my answer into this blog entry – apparently LinkedIn forums have a limit of 4000 characters per reply… (interestingly familiar number, by the way…:) So, I thought that it’s time to start writing my Oracle Exadata Performance series articles what I’ve planned for a while… with some war stories from the field, some stuff what I’ve overcome when researching for writing the Expert Oracle Exadata book etc.

Finding Oracle Homes which Oracle instances are using on Linux

Mon, 28 Feb 2011 12:42:23 +0000

I had a question about how to quickly identify which Oracle process runs out of which ORACLE_HOME on Linux. I have uploaded a little script for that – it’s basically looking up all PMON process IDs and then using /proc/PID/exe link to find out where is the oracle binary of a running process located. You may have to run this as root (as on some Linux versions I get “ls: cannot read symbolic link: Permission denied” error even when running this command as the owner of all Oracle homes (it seems to happen when your users UID and primary GID are different than thet setuid/setgid bits on the oracle binary):

New cursor_bind_capture_destination parameter in Oracle 11.2.0.2

Wed, 19 Jan 2011 23:33:09 +0000

I just noticed that there’s a new cursor_bind_capture_destination parameter in Oracle 11.2.0.2 (which is really more like Oracle 11gR3 version because of the large amount of new features in it, as opposed to just bugfixes). This parameter allows you to save some SYSAUX tablespace disk space – if the occasionally captured bind variable values (from V$SQL_BIND_DATA) take too much space. Normally these bind values (in a packed RAW form) are visible in DBA_HIST_SQLSTAT.

Performance Stories from Exadata Migrations

Wed, 12 Jan 2011 01:04:47 +0000

Here are my UKOUG 2010 slides about Exadata migration performance, this is real life stuff, not repeating the marketing material: Tanel Poder – Performance stories from Exadata Migrations View more presentations from tanelp.

Is this valid SQL syntax? :-)

Mon, 10 Jan 2011 23:07:56 +0000

I’m talking about this: select-1from from dual; Looks like invalid, right? Well, let’s run it: SQL> select-1from from dual; ROM ---------- -1.0E+000 This is because: Oracle doesn’t need whitespace for tokenizing the SQL statement (differences in character classes will do – as I’ve explained here) The first from “keyword” in the above statement is broken down to two tokens as an “F” right after a digit means that the preceding number is a FLOAT (and “D” means DOUBLE) and the tokenizer stops right there, knowing that whatever comes after this character (“ROM”) is a next token, which according to the Oracle SQL syntax rules will be assigned as the output column alias The following funky-looking SQL statements are also valid:

Snapper 3.52 – With Oracle 9.2 support!

Sun, 09 Jan 2011 19:28:00 +0000

As I promised last year, I have 2 christmas gifts for you. I have already forgotten what the other one was supposed to be :), but the first one is Snapper v3.52 which has (the much requested) Oracle 9.2 support! The syntax is the same, with Snapper you can now sample ASH-like data on Oracle 9.2 too. Instead of SQL_IDs it will display you SQL hash values: SQL> @snapper ash,ash1=user+sql_id,ash2=sid+event 5 1 all Sampling SID all with interval 5 seconds, taking 1 snapshots.

Asynch descriptor resize wait event in Oracle

Tue, 23 Nov 2010 21:01:05 +0000

A lot of people have started seeing “asynch descriptor resize” wait event in Oracle 11gR2. Here’s my understanding of what it is. Note that I didn’t spend too much time researching it, so some details may be not completely accurate, but my explanation will at least give you an idea of why the heck you suddenly see this event in your database. FYI, there’s a short, but incomplete explanation of this wait event also documented in MOS Note 1081977.

A little new feature for shared pool geeks :-)

Fri, 05 Nov 2010 01:22:53 +0000

If you’ve taken any shared pool dumps from 11g+ databases lately, you might have wondered about what all the memory allocation reason codes like SQLA^ea880c38, KGLS^da11791e might mean. Oracle 11g has introduced a little improvement in how library cache manager allocates shared pool chunks for its objects. Here’s an excerpt from a shared pool heap dump, but this is visible also from X$KSMSP (you should be very careful when thinking of running shared pool heap-dumps or querying X$KSMSP in busy production databases as they may hang your instance for a while).

COUNT STOPKEY operation (the where ROWNUM <= N predicate) doesn't process over ~4 Billion rows and returns wrong results

Mon, 25 Oct 2010 17:51:06 +0000

I was running tests on some very large tables on an Exadata cluster and found an interesting bug. Instead of having to query 4Billion row tables, I have reproduced this case with a cartesian join… Check this. I’m generating 8 Billion rows using a 3-way cartesian join of set of 2000 rows. So, this results in 2000 * 2000 * 2000 rows, which is 8 billion rows. SQL> with sq as (select null from dual connect by level <= 2000) select count(*) from sq a, sq b, sq c; COUNT(*) ---------- 8000000000 Everything worked well as you see.

Read currently running SQL statement’s bind variable values using V$SQL_MONITOR.BIND_XML in Oracle 11.2

Mon, 18 Oct 2010 11:11:29 +0000

The title pretty much says it. In Oracle 11.2, if you have the Diag+Tuning Pack licenses and the SQL monitoring kicks in for your SQL statement, then instead of the old fashioned ERRORSTACK dump reading you can just query the V$SQL_MONITOR.BIND_XML to find the values and metadata of your SQL statement’s bind variables. I’ve written an example here: https://tech.e2sn.com/oracle-living-books/oracle-troubleshooting/oracles-real-time-sql-monitoring-feature-vsql_monitor And a related comment – V$SQL_BIND_CAPTURE is not a reliable way for identifying the current bind variable values in use.

The most fundamental difference between hash and nested loop joins

Wed, 06 Oct 2010 20:00:23 +0000

Basically the most fundamental (or biggest or most important) difference between nested loop and hash joins is that: Hash joins can not look up rows from the inner (probed) row source based on values retrieved from the outer (driving) row source, nested loops can. In other words, when joining table A and B (A is driving table, B is the probed table), then a nested loop join can take 1st row from A and perform a lookup to B using that value (of the column(s) you join by).

Oracle Closed World presentation links

Mon, 20 Sep 2010 22:51:15 +0000

Thanks to everybody who attended my OCW hacking session! Sorry to guys who attended via webinar – I’ll do the session again in a few weeks, with audio from end to end hopefully! And I will get someone to assist me with monitoring the transmission quality and attendee questions etc. Note that this stuff is mostly for hacking and fun – don’t use the undocumented stuff in production! The links are below:

Which number takes more space in an Oracle row?

Thu, 02 Sep 2010 13:32:00 +0000

So, which number takes more bytes inside an Oracle row? A: 123 B: 1000000000000000000000000000000000000 And the correct answer is … (drumroll) … A! The “big” number 1000000000000000000000000000000000000 actually takes less space than the “small” 123! Let’s verify this: SQL> select vsize(123) A, vsize(1000000000000000000000000000000000000) B from dual; A B ---------- ---------- 3 2 WTF? Why does such a small number 123 take more space than 1000000000000000000000000000000000000 ? Well, the answer lies in how Oracle stores numbers.

Exadata v2 Smart Scan Performance Troubleshooting article

Fri, 30 Jul 2010 22:18:25 +0000

I finally finished my first Exadata performance troubleshooting article. This explains one bug I did hit when stress testing an Exadata v2 box, which caused smart scan to go very slow – and how I troubleshooted it: Troubleshooting Exadata v2 Smart Scan Performance Thanks to my secret startup company I’ve been way too busy to write anything serious lately, but apparently staying up until 6am helped this time! :-) Anyway, maybe next weekend I can repeat this and write Part 2 in the Exadata troubleshooting series ;-)

Dropping and creating tables in read only tablespaces?!

Sun, 11 Jul 2010 17:09:12 +0000

You probably already know that it’s possible to drop tables in Oracle read only tablespaces… (You did know that already, right? ;-) Here’s a little example: SQL> create tablespace ronly datafile '/u03/oradata/LIN112/ronly.01.dbf' size 10m; Tablespace created. SQL> create table test tablespace ronly as select * from all_users; Table created. SQL> alter tablespace ronly READ ONLY; Tablespace altered. SQL> drop table test; Table dropped. I just dropped a table from a read only tablespace!

The full power of Oracle’s diagnostic events, part 2: ORADEBUG DOC and 11g improvements

Wed, 23 Jun 2010 11:46:31 +0000

I haven’t written any blog entries for a while, so here’s a very sweet treat for low-level Oracle troubleshooters and internals geeks out there :) Over a year ago I wrote that Oracle 11g has a completely new low-level kernel diagnostics & tracing infrastructure built in to it. I wanted to write a longer article about it with comprehensive examples and use cases, but by now I realize I won’t ever have time for this, so I’ll just point you to the right direction :)

Oracle memory troubleshooting article

Fri, 28 May 2010 16:37:45 +0000

Randolf Geist has written a good article about systematic troubleshooting of a PL/SQL memory allocation & CPU utilization problem – and he has used some of my tools too! http://oracle-randolf.blogspot.com/2010/05/advanced-oracle-troubleshooting-session.html

Flexible Sqlplus command line history with RLWRAP

Fri, 07 May 2010 22:49:26 +0000

At Hotsos Symposium Training Day I used rlwrap with sqlplus – which gives nice command line editing and history capabilities for tools like sqlplus. Additionally I pre-generated commonly used Oracle keywords, data dictionary view and package names into rlwrap wordfile, so I got nice tab-completion too. Sqlplus sucks much less with rlwrap ;-) It’s relatively easy to install rlwrap on Unix (there are rlwrap RPMs out there, Solaris freeware packages and I installed it on Mac via macports.

Execution plan Quiz: Shouldn’t these row sources be the other way around ;-)

Tue, 27 Apr 2010 18:58:27 +0000

Here’s a little trick question. Check out the execution plan below. What the hell, shouldn’t the INDEX/TABLE access be the other way around?! Also, how come it’s TABLE ACCESS FULL (and not by INDEX ROWID) in there? This question is with a little gotcha, but can you come up with a query which produced such plan? ;-) ---------------------------------------------- | Id | Operation | Name | E-Rows | ---------------------------------------------- | 0 | SELECT STATEMENT | | | |* 1 | INDEX RANGE SCAN | PK_EMP | 1 | |* 2 | TABLE ACCESS FULL| EMP | 1 | ----------------------------------------------

Quiz: Explaining index creation

Fri, 23 Apr 2010 14:49:02 +0000

Did you know that it’s possible to use EXPLAIN PLAN FOR CREATE INDEX ON table(col1,col2,col3) syntax for explaining what exactly would be done when an index is created? That’s useful for example for seeing the Oracle’s estimated index size without having to actually create the index. You can also use EXPLAIN PLAN FOR ALTER INDEX i REBUILD to see whether this operation would use a FULL TABLE SCAN or a FAST FULL INDEX SCAN (offline index rebuilds of valid indexes can use this method).

cursor: pin S waits, sporadic CPU spikes and systematic troubleshooting

Wed, 21 Apr 2010 21:55:09 +0000

I recently consulted one big telecom and helped to solve their sporadic performance problem which had troubled them for some months. It was an interesting case as it happened in the Oracle / OS touchpoint and it was a product of multiple “root causes”, not just one, an early Oracle mutex design bug and a Unix scheduling issue – that’s why it had been hard to resolve earlier despite multiple SRs opened etc.

KGH: NO ACCESS – Buffer cache inside streams pool too!

Wed, 21 Apr 2010 19:21:44 +0000

Some time ago I wrote that since Oracle 10.2, some of the buffer cache can physically reside within shared pool granules. I just noticed this in an 11.2 instance: SQL> select * from v$sgastat where name like ‘KGH%’; POOL NAME BYTES ------------ -------------------------- ---------- streams pool KGH: NO ACCESS 4186144 So, it looks that also streams pool can surrender parts of its memory granules to buffer cache, if it’s unable to flush everything out from the granule for complete granule handover.

Non-trivial performance problems

Sat, 03 Apr 2010 10:07:26 +0000

Gwen Shapira has written an article about a good example of a non-trivial performance problem. I’m not talking about anything advanced here (such as bugs or problems arising at OS/Oracle touchpoint) but that sometimes the root cause of a problem (or at least the reason why you notice this problem now) is not something deeply technical or related to some specific SQL optimizer feature or a configuration issue. Instead of focusing on the first symptom you see immediately, it pays off to take a step back and see how the problem task/application/SQL is actually used by the users or client applications.

Calculate SQL_ID and SQL_HASH_VALUE from SQL text

Wed, 31 Mar 2010 17:39:05 +0000

Some time ago I wrote an article about the 10g+ SQL_ID being just a hash value of the SQL statement text. It’s just like the “old” SQL_HASH_VALUE, only twice longer (8 last bytes instead of 4 last bytes of the MD5 hash value of SQL text). Slavik Markovich has written a nice python script for calculating SQL_IDs and SQL hash values from SQL text using that approach. Slavik’s article is available here:

Oracle Latch Contention Troubleshooting

Sun, 28 Mar 2010 04:46:08 +0000

I wrote a latch contention troubleshooting article for IOUG Select journal last year (it was published earlier this year). I have uploaded this to tech.E2SN too, I recommend you to read it if you want to become systematic about latch contention troubleshooting: http://tech.e2sn.com/oracle/troubleshooting I’m working on getting the commenting & feedback work at tech.E2SN site too, but for now you can comment here at this blog entry…

Oracle Session Snapper v3.10

Mon, 22 Mar 2010 16:35:50 +0000

Hi all, long time no see! =8-) Now as I’m done with the awesome Hotsos Symposium (and the training day which I delivered) and have got some rest, I’ll start publishing some of the cool things I’ve been working on over the past half a year or so. The first is Oracle Session Snapper version 3! There are some major improvements in Snapper 3, like ASH style session activity sampling!

How to CANCEL a query running in another session?

Thu, 18 Feb 2010 04:14:54 +0000

Here’s an article of mine, explaining some of the Oracle internals and the OS touch point. It actually goes beyond explaining just commands for canceling SQL. I have written a couple of updates into the beginning of this article about SQL cancellation commands, but if you want to learn the internals, scroll down to The original article section. Update 1: As the beginning says, this article was meant as something interesting about Oracle internals and CTRL+C / OCICancel() handling.

New versions of LatchProf and LatchProfX for latch contention troubleshooting and tuning

Tue, 16 Feb 2010 03:01:51 +0000

The LatchProf and LatchProfX scripts allow you to be more systematic with latch contention troubleshooting and tuning. No more guesswork is needed as these scripts give you exact session IDs and in this version also SQLIDs of the troublemaking applications. You can download the new versions here: LatchProf (reads V$ views) LatchProfX (reads X$ tables, but gives better info, run as SYS) Example output (with SQLID info) is below: SQL> @latchprof name,sid,sqlid % % 100000 -- LatchProf 1.

Oracle Troubleshooting: How to read Oracle ERRORSTACK output?!

Sun, 14 Feb 2010 18:41:46 +0000

I have written the first article to the troubleshooting section of my new website tech.E2SN.com: It’s about a very valuable Oracle troubleshooting tool -> ERRORSTACK trace. I cover 4 frequently asked questions there: Reading the current executing SQL statement text from errorstack trace Reading the current executing PL/SQL package and PL/SQL source code line number from errorstack trace Reading the current bind variable values from errostack trace Identifying how much private (UGA) memory a cursor is using You can read it here:

Using Process Memory Matrix script for calculating Oracle process memory usage on Solaris

Thu, 11 Feb 2010 18:21:04 +0000

I just published a new script and article about calculating the real Oracle process memory usage on Solaris. The problem with V$PROCESS* views (and the V$SESSTAT) is that they will tell you what Oracle thinks it’s using, not the real amount of memory used. There will be discrepancies due how memory is actually allocated in OS, libraries out of Oracle’s control, the static memory areas inside Oracle binary and of course bugs.

Oracle Wait Event reference

Tue, 09 Feb 2010 10:35:57 +0000

Kyle Hailey has started putting together a much needed Oracle wait event reference. You can access it here. By the way, Oracle documentation also has a wait event reference section, it has more events, but it’s less detailed…

Oracle Peformance Visualization…

Wed, 03 Feb 2010 09:47:42 +0000

Coskan Gundogar and Karl Arao have written two interesting articles about Oracle performance analysis and visualization, check these out! Coskan’s article: http://coskan.wordpress.com/2010/01/27/working-with-statspack-part-1a-diagnosis/ Karl’s article: http://karlarao.wordpress.com/2010/01/31/workload-characterization-using-dba_hist-tables-and-ksar/ Note that in March I will be releasing PerfSheet v3.0, which will have lots of improvements! ;-)

Bind Variable Peeking – execution plan inefficiency

Tue, 02 Feb 2010 17:11:34 +0000

In my Beyond Oracle Wait interface article I troubleshooted a test case where an execution plan somehow went “crazy” and started burning CPU, lots of logical IOs and the query never completed. I have uploaded the test case that you can run here (note that it drops and creates tables T1..T5 in your schema): /ast/02_bind_peeking_nested_loops.sql Basically what I do is this: I run the query with bind variable values where only a handful of rows match the filter condition.

Recursive sessions…

Fri, 22 Jan 2010 04:43:29 +0000

I have published a new article to tech.e2sn.com about recursive sessions and ORA-00018: maximum number of sessions exceeded error message: http://tech.e2sn.com/oracle/oracle-internals-and-architecture/recursive-sessions-and-ora-00018-maximum-number-of-sessions-exceeded Note that I’m working on setting up RSS feed for tech.e2sn too, coming soon :)

Sometimes things are easy (Part 1): How to fix wrapped execution plan text?

Mon, 18 Jan 2010 16:26:33 +0000

What you see below is a common problem. Someone sends you (or posts to a forum) a wide execution plan, which is unreadable because of wrapped lines. For example, this one below: -------------------------------------------------------------------------------- ------------------- | Id | Operation | Name | E-Rows | OMem | 1Mem | Used-Mem | -------------------------------------------------------------------------------- ------------------- | 0 | SELECT STATEMENT | | | | | | | 1 | SORT AGGREGATE | | 1 | | | | |* 2 | HASH JOIN | | 13 | 1102K| 1102K| 355K (0)| |* 3 | HASH JOIN | | 13 | 988K| 988K| 367K (0)| |* 4 | HASH JOIN | | 13 | 921K| 921K| 621K (0)| |* 5 | HASH JOIN OUTER | | 13 | 836K| 836K| 1224K (0)| |* 6 | HASH JOIN | | 13 | 821K| 821K| 501K (0)| |* 7 | HASH JOIN | | 13 | 1102K| 1102K| 501K (0)| | 8 | MERGE JOIN CARTESIAN| | 1 | | | | |* 9 | TABLE ACCESS FULL | PROFILE$ | 1 | | | | | 10 | BUFFER SORT | | 1 | 73728 | 73728 | | |* 11 | TABLE ACCESS FULL | PROFILE$ | 1 | | | | |* 12 | TABLE ACCESS FULL | USER$ | 36 | | | | | 13 | TABLE ACCESS FULL | PROFNAME$ | 1 | | | | |* 14 | TABLE ACCESS FULL | RESOURCE_GROUP_MAPPING$ | 1 | | | | | 15 | TABLE ACCESS FULL | TS$ | 7 | | | | | 16 | TABLE ACCESS FULL | TS$ | 7 | | | | | 17 | TABLE ACCESS FULL | USER_ASTATUS_MAP | 9 | | | | -------------------------------------------------------------------------------- ------------------- So now you either try to manually edit and fix the execution plan text so you could read it or ask the developer to send the execution plan again.

Beyond Oracle Wait Interface – Part 2

Fri, 15 Jan 2010 10:53:57 +0000

This is the second part of the joint blog “project” with James Morle, called “The Wait Interface Is Useless (Sometimes)”. We already did a joint presentation on this topic at UKOUG and more conferences will follow :) Read the first part by James here for intro. So, where do we go when Oracle’s wait interface doesn’t help us? We will show multiple techniques over time, but here’s where I normally continue when wait interface is “useless”.

NULL is not zero!

Wed, 30 Dec 2009 06:48:15 +0000

Some time ago I wrote a post about how COUNT(*) and COUNT(column) are semantically different things (link). Such queries may return different results if the column counted has NULLs in it. And the difference comes from that NULL is not a value, it’s rather a state which says “value unknown” or “no value entered”. So, you better understand how NULLs interact with your SQL constructs if you call yourself a DBA or a database developer ;-)

Measuring what matters

Wed, 23 Dec 2009 00:58:39 +0000

Cary Millsap’s recent post prompted me to write down some of the related thoughts in my head. Here are few of my mantras for systematic troubleshooting and performance tuning, which have materialized in my head over the years of work: Picking the right starting point to troubleshooting and performance tuning is the most important decision in that process. Pick the wrong starting point and you end up going in circles. The scope of your performance data needs to match the scope of your problem, otherwise you end up going in circles.

Finding the reasons for excessive logical IOs

Thu, 19 Nov 2009 17:16:38 +0000

There’s another interesting thread going on in Oracle-L, about understanding logical IOs and drilling down into their reasons. Of course sometimes (or rather usually) the excessive logical IOs come from a bad execution plan (when a nested loop loops over lots of datablocks again and again or a wrong index is used for driving a query etc), but sometimes the excessive LIOs are caused by some internal issues, like space management etc.

A convenient tool I use for reporting logical IO reasons is (again) my Snapper! It has an option “b” for reporting Buffer get reasons or as I use below – option “a” shows All information Snapper can show.

There are couple of gotchas though which make this approach imperfect:

The X$ tables Snapper uses for LIO reason reporting contain instance-wide counters, not specific to a single testing session. Thus you either need to be the single user in your database when experimenting and even then the background activity may increment some counters while you are testing too. I have sometimes suspended all other processes (kill -STOP and kill -CONT to resume) or used Flash Freeze (oradebug ffbegin and ffresumeinst) to hang the whole instance that there would be no other activity going on.
These buffer get reason counters are not maintained properly in Oracle 11g, probably due an optimization effort and some changes for faster pinning of buffer cache buffers (there’s a parameter called _fastpin_enable which is set to 1 in 11g and it enables so called fastpath buffer gets. If you see v$sesstat statistics such “consistent gets from cache (fastpath) or “db block gets from cache (fastpath)” being inremented, then fastpath buffer gets/pins are used. Note that I do have a script which works also on 11g but I’ll write about that one some time in the future :)

Anyway, if you are testing in an environment exclusively used by you, on Oracle 10.2 or lower, then you can run snapper with the gather=a option to report a bunch instance-level statistics in addition to the standard session-level stats:

BUFG – Buffer get reasons (both consistent and current mode gets)
LATG – Latch gets (both willing to wait and immediate gets)
ENQG – Enqueue gets

Here’s an example, prepare for long output:

Explain Plan For command may show you the wrong execution plan – Part 1

Wed, 18 Nov 2009 02:33:24 +0000

In Oracle-L mailing list a question was asked about under which conditions can the explain plan report a wrong execution plan (not the one which was actually used when a problem happened). I replied this with the following, but thought to show an example test case of this problem too: The optimizer statistics the EXPLAIN PLAN ends up using are different from the statistics the other session ended up using

Detect chained and migrated rows in Oracle – Part 1

Wed, 04 Nov 2009 22:34:47 +0000

I received a question about migrated rows recently.

It was about how to detect migrated rows in a 200TB data warehouse, with huge tables – as the ANALYZE TABLE xyz LIST CHAINED ROWS INTO command can not be automatically parallelized at table level (as DBMS_STATS can be, but oh, DBMS_STATS doesn’t gather the migrated/chained row info). Therefore the analyze command would pretty much run forever before returning (and committing) the chained row info in the output table. Also as there are regular maintenance jobs running on these tables (I suspect partition maintentance for example), then it wouldn’t be nice to keep running ANALYZE on the whole table constantly.

So, is there any faster or better way for finding the amount of migrated rows?

Ihave two answers to this.

Answer 1:

As we are dealing with a huge 200+ TB data warehouse its tables/indexes are most likely partitioned. Thus you could use the ANALYZE TABLE xyz PARTITION (abc) LIST CHAINED ROWS command to analyze individual partitions, even in parallel (sqlplus sessions) if you like. This would allow you to focus only on the partitions of interest (the latest ones, with the heaviest activity perhaps).

SQL> create table CHAINED_ROWS (
2    owner_name         varchar2(30),
3    table_name         varchar2(30),
4    cluster_name       varchar2(30),
5    partition_name     varchar2(30),
6    subpartition_name  varchar2(30),
7    head_rowid         rowid,     -- actual chained row's head piece address in the segment
8    analyze_timestamp  date
9  );

Table created.

SQL>
SQL> analyze table tmp partition (sys_p501) list chained rows; -- the default table name used for output is "CHAINED_ROWS"

Table analyzed.

SQL> analyze table tmp partition (sys_p502) list chained rows;

Table analyzed.

SQL> select partition_name, count(*) from chained_rows group by partition_name;

PARTITION_NAME                   COUNT(*)
------------------------------ ----------
SYS_P502                              252
SYS_P501                             5602

SQL>

So, from above you see its possible to find out partition (or even sub-partition level row chaining).

However this above command lists you both CHAINED rows and MIGRATED rows (even though Oracle calls them all chained rows internally, as the chaining mechanism is the same for both cases).

Chained row is a row which is too large to fit into a block, so will always have to be split between multiple different blocks – with an exception of intra-block chaining which is used for rows with more than 255 columns. Migrated row on the other hand is a row which has been updated larger than it initially was – and if as a result it doesn’t fit into its original block, the row itself is moved to a new block, but the header (kind of a stub pointer) of the row remains in original location. This is needed so that any indexes on the table would still be able to find that row using original ROWIDs stored in them). If Oracle didn’t leave the row head piece in place then it would always go and update all indexes which have the ROWID of the migrating row in them.

Why should we care whether a row is a real chained row or just a migrated row?

It’s because if the row is chained, then any reorgs would not help you – if a row is too big to fit into a block, its too big to fit into a block no matter how many times you move around the table. (Note that if you have large tables full of rows longer than 8KB there’s likely something wrong with your design).

But migrated rows on the other hand are “chained” into another block due some update which made them not fit into existing block anymore. This happens when PCTFREE is set too low compared to real row growth factor and sometimes you may want to fix it by reorganizing the table/partition with ALTER TABLE/PARTITION MOVE or by backing the rows up, deleting them and reinserting them back to the table (that one makes sense when only a small amount of rows in a table are migrated).

If you are completely sure that you don’t have any rows longer than the free space in an empty block (thus all individual rows would fit into a block and would need to be split among multiple blocks) then you can conclude that all the rows reported were migrated due their growth.

Another option would be to query out all or a sample of these chained/migrated rows and actually measure how long they are if all columns are put together. This could be done using vsize() function (or also dump() and lengthb() in some cases). Of course the column and row header overhead would need to be accounted in as well.

So, this already gets pretty complex and there are more tiny details which we should take into account… thus I will introduce another way to look into the row migration/chaining thing:

**Answer 2: (Alternatively called “should we care?”)

What’s a good way to learn some Oracle internals every day?

Mon, 26 Oct 2009 16:11:10 +0000

Sometimes when an attendee describes me some totally weird problem during a seminar, I am immediately able to answer something like “Hey this looks like a bug related to this Oracle configuration and can be influenced by xyz”.

And then people ask me “How the hell do you know all this stuff?”

Well, I haven’t been bitten by all of these bugs myself, but I have been doing something for many years, almost every day… reading my email!

Oh, and additionally I have configured Metalink to send me daily updates about new/updated notes, forum articles and… bug descriptions!

The last part is very important. Bug descriptions tell you something about new bugs found (and old bugs rediscovered) and sometimes their details tell you an interesting piece or two about Oracle internals related to them.

SystemTap is production supported in Redhat EL5.4

Sun, 25 Oct 2009 15:53:45 +0000

If you don’t know what SystemTap is – it’s the Linux world’s attempt to build Solaris DTrace style safe dynamic instrumentation into Linux kernel. I’m not going into religious discussions which one is better here, I have used both SystemTap and DTrace successfully for diagnosing low level issues inside OS kernel, so both are good enough for me :) The problem with SystemTap though has been that it’s not production quality, it’s rather been a technology preview.

KGL simulator, shared pool simulator and buffer cache simulator – what are these?

Mon, 14 Sep 2009 00:13:39 +0000

If you have queried v$sgastat on recent Oracle versions (by which I mean 9i and above) you probably have seen allocations for some sort of simulators in Oracle instance. Here’s an example:

SQL> select * from v$sgastat where lower(name) like '%sim%' order by name;

POOL         NAME                            BYTES
------------ -------------------------- ----------
shared pool  kglsim alloc latch area          1700
shared pool  kglsim alloc latches               68
shared pool  kglsim count of pinned he        9248
shared pool  kglsim free heap list             204
shared pool  kglsim free obj list              204
shared pool  kglsim hash table                4104
shared pool  kglsim hash table bkts        2097152
shared pool  kglsim heap                    635536
shared pool  kglsim latch area                1700
shared pool  kglsim latches                     68
shared pool  kglsim main lru count           87040
shared pool  kglsim main lru size           174080
shared pool  kglsim object batch            909440
shared pool  kglsim pin list arr               816
shared pool  kglsim recovery area             2112
shared pool  kglsim sga                      22092
shared pool  kglsim size of pinned mem       18496
shared pool  ksim client list                   84
shared pool  log_simultaneous_copies           480
shared pool  sim cache nbufs                   640
shared pool  sim cache sizes                   640
shared pool  sim kghx free lists                 4
shared pool  sim lru segments                  640
shared pool  sim segment hits                 1280
shared pool  sim segment num bufs              640
shared pool  sim state object                   48
shared pool  sim trace buf                    5140
shared pool  sim trace buf context             120
shared pool  sim_knlasg                       1200
shared pool  simulator hash buckets          16512
shared pool  simulator latch/bucket st        4608

31 rows selected.

See, a bunch of “kgl sim” and then just “sim” allocations.

… or sometimes you can see latch contention on following latches:

SQL> select name from v$latch where name like '%sim%';

NAME
-------------------------------------------------------
ksim membership request latch
simulator lru latch
simulator hash latch
sim partition latch
shared pool simulator
shared pool sim alloc

6 rows selected.

Again, there seems to be some “simulation” work going on in Oracle instance.

So what are these simulators about?

KGH: NO ACCESS allocations in V$SGASTAT – buffer cache within shared pool!

Wed, 09 Sep 2009 15:05:49 +0000

Since Oracle 10.2 it’s valid to say that buffer cache can be stored inside shared pool.

Now you may think I’m crazy, but read until the end of the post – no matter how crazy I may sound – I have proof!

Here it is:

Few years ago I started noticing a strange memory allocation in shared pool (in V$SGASTAT), called KGH: NO ACCESS.

SQL> select * from v$sgastat where name = 'KGH: NO ACCESS';

POOL         NAME                            BYTES
------------ -------------------------- ----------
shared pool  KGH: NO ACCESS               10513696

SQL>

You see, some 10 MB of memory in shared pool has been allocated for something called KGH: NO ACCESS.

Ok, lets see where this memory resides inside shared pool. We can use x$ksmsp view for that, this view has a line in it for each chunk of memory allocated from it (and also the free chunks), along the reasons (or comments) for what reason these chunks were allocated.

Oracle 11gR2 has been released – and with column oriented storage option

Tue, 01 Sep 2009 18:07:05 +0000

You may already have noticed that Oracle 11gR2 for Linux is available for download on Oracle.com website, with documentation. And this document ends speculation about whether Oracle 11.2 will support column-oriented storage – yes it will: http://www.oracle.com/technology/products/database/oracle11g/pdf/oracle-database-11g-release2-overview.pdf However, this is apparently available on Exadata storage only as a new error message below indicates: ORA-64307: hybrid columnar compression is only supported in tablespaces residing on Exadata storage Cause: An attempt was made to use hybrid columnar compression on unsupported storage.

latch: cache buffers chains latch contention – a better way for finding the hot block

Thu, 27 Aug 2009 09:37:08 +0000

Here's a treat for Oracle performance professionals and geeks who are looking for more systematic ways for cache buffers chains (CBC) latch contention troubleshooting. Cache buffers chains latches are taken when a process wants to walk through a cache buffer hash chain, looking if the block with required DBA (data block address) is in buffer cache. If the block happens to be in cache, then in most cases it has to be pinned first before use and unpinned after use, to make sure no-one else can perform an incompatible operation on that block at the same time.

Select COUNT(*) and COUNT(column) are different things!

Sat, 22 Aug 2009 03:58:19 +0000

Every now and then I see someone wondering why Oracle is “returning wrong results” for some count queries when counting using COUNT(column_name) instead of COUNT(*) or COUNT().

Oracle is actually returning correct results, its just that sometimes the people asking the questions haven’t realized that COUNT(column) is something semantically different from COUNT(*).

COUNT(*) operation counts all rows fed to it by execution plan branch under it.

COUNT(column) operation on the other hand counts all non-null values in that column from rows fed to it by execution plan branch under it.

And here’s a little example:

Alter system kill session and ORA-00031: session marked for kill

Thu, 13 Aug 2009 07:22:27 +0000

I’m sure you are all familiar with this situation:

SQL> alter system kill session '152,33';
alter system kill session '152,33'
*
ERROR at line 1:
ORA-00031: session marked for kill

The session trying to issue the kill will hang for 60 seconds and then return this “session marked for kill” message. And the target session does not get killed at all.

So why is that?

The issue is in what this alter system kill command is doing. It’s not actually killing the target session (like kill -9 would do for OS processes). It just sets a bit in the target sessions state object, which marks that the target session should end. But its entirely up the target session to check this bit and act on it!

So, intead of ALTER SYSTEM KILL SESSION, the command should look something like ALTER SYSTEM ASK SESSION TO COMMIT SUICIDE.

All the kill session command is doing is ASK the target session to clean up and exit – via setting that bit.

Now, normally the target sessions are nice and check that bit often enough in their code, act on it and die.

But sometimes when the target session happens to be busy looping in some tight loop (due a bug perhaps) or is hung, then it never gets to check that “please die” bit and never exits.

An interview with me

Mon, 10 Aug 2009 01:56:46 +0000

Iggy Fernandez posted an interview with me (published in NoCOUG journal) on his blog. If you are interested in a little bit of my history and some more general (not-so-technical) thoughts of mine, check it out here: http://iggyfernandez.wordpress.com/2009/08/08/great-expectations-an-interview-with-tanel-poder/

How to detect when a cursor was closed from SQL trace output?

Thu, 09 Jul 2009 11:50:07 +0000

After Randolf’s comment on my last post about identifying cursor SQL text from sql trace file I think one thing needs elaboration. I mentioned earlier in this post that this cursor dumping technique works “as long as the cursor of interest is still open”. So how do you know whether this cursor of interest is still open or has been closed and that slot reused by some other statement instead? You would not want to get misled to wrong SQL statement…

Identify the SQL statement causing those WAIT #X lines in a (top-truncated) sql tracefile

Thu, 09 Jul 2009 09:44:40 +0000

Have you experienced such situation before?

A performance issue happens in production – let say some batch job has ran way over time
You enable SQL trace on the problem session (while the problem is already ongoing)
In tracefile you see lots of waits (or execs or fetches) caused by cursor X
You grep for “PARSING IN CURSOR #X” above the waits in the tracefile but don’t find the corresponding parsing entry nor SQL text there (this is a “top-truncated” tracefile)
You really want to know which SQL corresponds to all those WAIT #X/FETCH #X lines

An example output would be here:

WAIT #2: nam='db file scattered read' ela= 25190 file#=1 block#=863 blocks=2 obj#=113 tim=548703769817
WAIT #2: nam='db file sequential read' ela= 51397 file#=1 block#=864 blocks=1 obj#=113 tim=548719015123
WAIT #2: nam='db file scattered read' ela= 348553 file#=1 block#=5969 blocks=8 obj#=113 tim=548732315966
WAIT #2: nam='db file scattered read' ela= 32275 file#=1 block#=5972 blocks=5 obj#=113 tim=548771073619

In above case the cursor #2 was causing all these WAIT lines. The problem is, that if the cursor #2 was parsed before the SQL trace was enabled, we won’t have the SQL statement dumped in the tracefile! Thus, by looking into the tracefile only, there’s no way to know which exact SQL statement is causing those waits.

If this was a single long-running SQL statement, you could easily look into v$session.sql_hash_value and map this to SQL text using v$sql. However things aren’t as simple when you have lots of PARSE,EXEC,FETCH calls of other statements happening in between (as the sql_hash_value would be constantly changing due each such call.

Also, other v$ (or x$) views don’t show you anything about the WAIT #X mapping to SQL statement as this stuff is not stored in SGA.

As a workaround, you could take the obj# and map it back to object number the (io) wait happened on and make some guesses about SQL statement from there. Or you could dump the datablock using file# and block# numbers, look at the data and make some further guesses about SQL statement, but guesses are no good as they can be wrong and misleading – and you would be dependent on luck in your troubleshooting. But you want to be systematic, right? ;-)

So, in case of the top-truncated tracefile, if you want to be sure which WAIT# (or FETCH#,EXEC#) lines correspond to which SQL statement, we need to use some other way than v$views or trying to guess based on bits in the tracefile.

(Secret) Preview of Oracle 12g CBO leaked from Oracle labs…

Sat, 27 Jun 2009 07:13:54 +0000

After doing my seminar in Spain last month, one of the attendees sent me a secret note about what the CBO would look like in Oracle 12g. Apparently it’s re-engineered from ground and completely different from anything we’ve seen before. It goes beyond being automatic, intelligent and auto-tuning, it’s actually edible too! The New CBO Thanks to John Ospino for sending me this insider-info ;)

Oracle memory troubleshooting, Part 3: Automatic top subheap dumping with heapdump

Wed, 24 Jun 2009 12:31:07 +0000

If you haven’t read them – here are the previous articles in Oracle memory troubleshooting series: Part 1, Part 2

In Oracle, the HEAPDUMP dump event in Oracle allows you to dump various heap contents to tracefile. With adding the “level” parameter to this dump event, you can specify which heaps to dump.

Julian Dyke has documented most of the levels here.

There are two little known, but useful level bits for heapdumps – bit 0x10000000 and 0x20000000. These allow Oracle to dump top-5 biggest subheaps in a heap recursively.

When bit 0x10000000 is enabled then Oracle dumps the top-5 subheaps inside any heap its instructed to dump.

When bit 0x20000000 is enabled then Oracle dumps the top-5 subheaps as mentioned above, but in addition Oracle recursively dumps top-5 subheaps inside any subheaps automatically dumped above. So instead of dumping 1 level of subheaps, Oracle recursively dumps 2 levels if such sub-sub-heaps exist.

This reduces the amount of manual work – as Oracle can drill down towards the root cause automatically and dump the relevant information.

Here’s a little test case:

I set the serverout buffer unlimited (Oracle 10.2+) so that Oracle would buffer unlimited amount of dbms_output data in UGA (this is also a “good” way for using up all the memory in your server so you use “unlimited” with care).

SQL> set serverout on size unlimited
SQL>
SQL> exec for i in 1..1000000 loop dbms_output.put_line(lpad('x',10,'x')); end loop;

Without the recursive top subheap dumping we would see output like this (after processing the tracefile with heapdump_analyzer):

This is the usual way for dumping a target process private heap:

SQL> oradebug setorapid 35
Oracle pid: 35, Unix process pid: 26457, image: oracle@linux03
SQL> oradebug dump heapdump 1     <-- level 1 dumps all top level private heaps (top-level PGA,UGA and call heap)
Statement processed.
SQL>

And the output is:

[oracle@linux03 trace]$ heapdump_analyzer LIN11G_ora_26486.trc

  -- Heapdump Analyzer v1.00 by Tanel Poder (  )

  Total_size #Chunks  Chunk_size,        From_heap,       Chunk_type,  Alloc_reason
  ---------- ------- ------------ ----------------- ----------------- -----------------
    55065316     841      65476 ,     top uga heap,         freeable,  session heap          <-- session heap is allocated from top uga heap
    41218392    2517      16376 ,     session heap,         freeable,  koh-kghu sessi      <-- koh-kghu session heap is allocated from session heap
    13650208     836      16328 ,     session heap,             free,
       65520       1      65520 ,         pga heap,             free,
       65476       1      65476 ,     top uga heap,         recreate,  session heap
       57736      14       4124 ,     session heap,         freeable,  kxsFrame4kPage
...

We would see that most of the uga memory (roughly 41MB of 55MB) is allocated for for some reason “koh-kghu sessi”. This is a session heap where from allocations for various objects like PL/SQL variables, records and collections are done. So when we’d want to drill down and see inside that heap we could use the HEAPDUMP_ADDR dump with that heap descriptors address as parameter to look inside it.

However with these extra bits mentioned above, Oracle can automatically dump us the contents of biggest subheaps inside the heaps we asked it to dump:

Using Perfsheet and TPT scripts for solving real life performance problems

Tue, 09 Jun 2009 22:27:52 +0000

Karl Arao has written a nice blog entry where he used also some of my tools for visualizing and understanding performance of a RAC cluster suddenly gone slow. He presents a systematic approach he used for gathering evidence and also hopefully you notice that having the ability to easily visualize performance data (with PerfSheet for example) can be very helpful and time saving when troubleshooting non-trivial problems. Check his blog entry out here:

ORA-04031 errors and monitoring shared pool subpool memory utilization with sgastatx.sql

Thu, 04 Jun 2009 22:24:11 +0000

Since Oracle 9.2 the shared pool can be “partitioned” into multiple parts. This was probably done for relieving shared pool latch contention for crappy applications (which use shared pool latches too much due bad cursor or connection management).

The “partitions” are called shared pool subpools and there can be up to 7 subpools. Each subpool is protected by a separate shared pool latch and each subpool has its own freelists and LRU list. If you are interested in more details, a good starting point is this whitepaper by Oracle.

There are few different ways for detecting how many subpools you have in use. The more convenient ones are here:

You could query X$KGHLU which has a line for each shared pool subpool and (from 10g) also java pool if it’s defined:

SQL> select count(distinct kghluidx) num_subpools
  2  from x$kghlu
  3  where kghlushrpool = 1;

NUM_SUBPOOLS
------------
           7

The “kghlushrpool” column, which is 1 for shared pool subheaps and 0 for java pool, isn’t there in 9i (and in 9i the java pool apparently is not reported in x$kghlu anyway).

The reason why I don’t just count all matching lines from x$kghlu but use count distinct instead is that in Oracle 10.2.0.1 there are 4x more lines reported in this x$table. There’s an additional concept called sub-sub-pool starting from 10.2 where each shared pool sub-pool is split futher into 4 areas (allocations with different expected lifetime/durations go into different sub-sub-pools, but the same sub-pool latch protects all activity in sub-sub pools too). But in 10.2.0.1 the x$kghlu reports all sub-sub-pools too for some reason. The whitepaper from Oracle mentioned above explains this in more detail.

So from above output I see that in my instance all 7 shared pool subpools are in use. Oracle determines the number of needed subpools (during instance startup) based on your shared pool size and cpu_count. IIRC in 9.2 if you had 4 CPUs or more AND the shared_pool_size was bigger than 256 MB then 2 subpools were used, in 10g shared_pool_size had to be bigger for that, 512 MB I think and in 11g its 1GB. I don’t recall the exact threshold values and that’s not really important as you can see yourself how many subpools are in use with the above query.

For sake of this experiment I set the _kghdsidx_count variable to 7, this parameter can be used to force the number of subpools you want. In 9.2 days it was actually quite common to set this back to 1 IF you had ORA-4031 errors AND the reason was diagnosed to be free space imbalance between subpools. However since 10g this has been almost unnecessary as Oracle has improved their heap management algorithms.

SQL> @pd kghdsidx

NAME                                          VALUE                          DESCRIPTION
--------------------------------------------- ------------------------------ ------------------
_kghdsidx_count                               7                              max kghdsidx count

The script above queries few X$ tables to show the value of this hidden parameter.

So far the two above approaches have required access to X$ tables which usually means you need to be logged on as SYSDBA. What if you don’t have such access?

In such case you can work this out pretty reliably by looking into how many of the shared pool latches are actually in use. All 7 latches are always there, even if you have less subpools in use, that number is hardcoded into Oracle. But you can see how many latches have a significant number of gets against them.

In my case its evident that all latches are in use, they all have significant number of gets against them:

Oracle Performance Visualization videos from Sydney

Wed, 27 May 2009 13:18:31 +0000

Alex Gorbachev invited me to speak at Oracle Meetup @ Sydney event in April. I in addition to useful (and less cool) stuff I showed some useful (and very cool) stuff there too. I demoed my PerfSheet tool and also something completely new what I’ve been working on for last few months. Alex has uploaded couple of videos in to youtube, if you’re interested what I’ve been up to in last few months, check them out here:

Scripts for showing execution plans via plain SQL and also in Oracle 9i

Tue, 26 May 2009 20:42:05 +0000

Hi all,

Here are few scripts which allow you to query SQL execution plans and their execution statistics out from V$SQL_PLAN and V$SQL_PLAN_STATISTICS yourself.

Why would you want to do this as there’s the DBMS_XPLAN.DISPLAY_CURSOR() function in Oracle 10g?

Well, my scripts work also on Oracle 9.2. Also they give you better detail (and flexibility) compared to DBMS_XPLAN.DISPLAY_CURSOR.

Of course in 9i there’s the DBMS_XPLAN.DISPLAY function which you can use in conjuction with EXPLAIN PLAN FOR command, but that approach has problems. EXPLAIN PLAN treats all bind variables as varchar2 datatype, possibly causing the reported execution plan to be different from reality. Also, if you have a long running report from last night and you run explain plan today morning, then if optimizer statistics have changed meanwhile, the explain plan command might come up with a different execution plan again, which is different from reality.

So that’s why in 10g we have the DBMS_XPLAN.DISPLAY_CURSOR which goes directly to required library cache child cursor and extracts (unparses) the execution plan from there. The function uses V$SQL_PLAN% views as its data source. And guess what – these views are there in version 9.2 already! And thats’ where my scripts come in:

Here’s an example. Let’s set statistics_level=all so we get rowsource level execution stats for the cursor (note that this parameter makes your query consume much more CPU so it should only be used at session level for troubleshooting a specific performance issue):

SQL> alter session set statistics_level = all;

Session altered.

SQL> select count(*) from all_users;

  COUNT(*)
----------
        36

I know the hash value of this query, so lets report its execution plan, directly from library cache. This is the REAL execution plan inside that child cursor, not some estimate like EXPLAIN PLAN command gives:

“Free” DBA_HIST AWR views in 11g…

Wed, 06 May 2009 15:36:12 +0000

I just noticed this in 11g Licensing doc ( http://download.oracle.com/docs/cd/B28359_01/license.111/b28287/options.htm#sthref69 ): All data dictionary views beginning with the prefix DBA_HIST_ are part of this pack, along with their underlying tables.The only exception are the views: DBA_HIST_SNAPSHOT, DBA_HIST_DATABASE_INSTANCE, DBA_HIST_SNAP_ERROR, DBA_HIST_SEG_STAT, DBA_HIST_SEG_STAT_OBJ, and DBA_HIST_UNDOSTAT. They can be used without the Diagnostic Pack license. This exception is not present in 10.2 license guide, so before 11g you can query V$SEGMENT_STATISTICS and V$UNDOSTAT’s history “for free” :)

I have been troubleshooting since I was a kid! :)

Fri, 01 May 2009 13:02:57 +0000

People sometimes ask how come I know so much about Oracle (and some Unix) internals and how do I have such a passion for these things. Also, another thing you may have noticed is that for some reason a large amount of hardcore computer hackers, low level troubleshooters and various internals experts happen to come from Russia. And I can answer why this is so! A little bit of history: I am an Estonian, was born in Estonia and lived over 20 of my first years there.

Tracing Oracle SQL plan execution with DTrace

Fri, 24 Apr 2009 06:11:48 +0000

SQL is a declarative language – in other words you just declare what needs to be done and Oracle takes care of the part how it’s done.

However there’s nothing declarative about the actual SQL execution when it happens. SQL plan is just a tree of kernel rowsource functions executed in a specific order (defined in child cursor’s sql area).

The root of SQL plan is where the fetch function (opifch2 for example) gets the rows for passing back to the user (or PL/SQL engine).

The branches are operations like joins, union etc, which don’t have access to any data themselves and can just call other functions recursively to get rows

The leaves are the execution plan operations without any children, they call data layer to acces actual datablocks.

The first execution plan line (with lowest ID) without any children is the one where data access starts, that’s the place where first logical IO happens.

A commented exec plan is below:

SQL> select count(*) from all_users;

  COUNT(*)
----------
        35

SQL> @x

PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------
SQL_ID  b2zqhgr5tzbpk, child number 0
-------------------------------------
select count(*) from all_users

Plan hash value: 3268326079

--------------------------------------------------------------------------
| Id  | Operation            | Name  | E-Rows |  OMem |  1Mem | Used-Mem |
--------------------------------------------------------------------------
|   1 |  SORT AGGREGATE      |       |      1 |       |       |          | <- ROOT
|*  2 |   HASH JOIN          |       |     35 |  1517K|  1517K|  637K (0)|   <- BRANCH
|*  3 |    HASH JOIN         |       |     35 |  1593K|  1593K| 1361K (0)|     <- BRANCH
|   4 |     TABLE ACCESS FULL| TS$   |     13 |       |       |          |       <- LEAF
|*  5 |     TABLE ACCESS FULL| USER$ |     35 |       |       |          |       <- LEAF
|   6 |    TABLE ACCESS FULL | TS$   |     13 |       |       |          |     <- LEAF
--------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - access("U"."TEMPTS#"="TTS"."TS#")
   3 - access("U"."DATATS#"="DTS"."TS#")
   5 - filter("U"."TYPE#"=1)

I have written about how to map execution plan lines back to kernel functions here:

</2008/06/15/advanced-oracle-troubleshooting-guide-part-6-understanding-oracle-execution-plans-with-os_explain/>

The above approach is based on pstack, mostly useful for demonstrations but has helped me to diagnose one spinning condition in an execution plan once (that’s the whole reason I came up with this technique).

As I said above, SQL execution just means that the kernel’s rowsource functions are executed in a loop with order and hierarchy specified in the child cursor’s execution plan.

So, if you want to learn and really understand the sequence of SQL plan execution – it’s dead easy with DTrace. Here’s what happens when you fetch from the above execution plan:

Seminar feedback and pictures from Singapore

Mon, 20 Apr 2009 10:54:43 +0000

I’ve been busy with a series of seminars, so haven’t managed to blog much… Karl Arao has posted some feedback from my Singapore seminar and also the pictures we took with attendees. By the way, he has other good Oracle stuff in his blog so check it out here.

Read OS environment variables using DBMS_SYSTEM.GET_ENV()

Fri, 27 Mar 2009 17:28:58 +0000

Check out this article by Vikram Das about how to read OS environment variables using PL/SQL. I did not know that!

I’m an Oracle ACE Director now :)

Fri, 27 Mar 2009 00:46:45 +0000

Many people have asked me that how come I’m not an Oracle ACE yet. From this week I am an Oracle ACE Director. Here’s a link to my profile. Its pretty big honor to be recognized by Oracle Corp at such high level. I started working with Oracle software in 1997 and got really interested in that stuff after realizing how powerful the database was and how complex some internal details could be.

Oracle 11g: Reading alert log via SQL

Sun, 22 Mar 2009 01:38:15 +0000

Oracle has done some major improvements in the diagnosability infrastructure in version 11g. Here’s one little detail.

Before Oracle 11g it is possible to access the alert log via SQL using an external table or a pipelined function which in turn uses utl_file.

After reading the text you need to parse it to extract the information you need from there.

Starting from 11g Oracle does all this work for you. There is a fixed table X$DBGALERTEXT, when you query it, Oracle reads the log.xml from alert directory (which contains all the data what alert.log does), parses it and returns the details back as rows:

SQL> select message_text from X$DBGALERTEXT where rownum <= 20;

MESSAGE_TEXT
-----------------------------------------------------------------------------------------------------------------
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Shared memory segment for instance monitoring created
Picked latch-free SCN scheme 2
Using LOG_ARCHIVE_DEST_10 parameter default value as USE_DB_RECOVERY_FILE_DEST
Autotune of undo retention is turned on.
IMODE=BR
ILAT =18
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up ORACLE RDBMS Version: 11.1.0.7.0.
Using parameter settings in client-side pfile /u01/app/oracle/admin/LIN11G/pfile/init.ora on machine linux03
System parameters with non-default values:
  processes                = 150
  memory_target            = 404M
  control_files            = "/u01/oradata/LIN11G/control01.ctl"
  control_files            = "/u01/oradata/LIN11G/control02.ctl"
  control_files            = "/u01/oradata/LIN11G/control03.ctl"
  db_block_size            = 8192

20 rows selected.

This is the text representation, but you can get individual details from other columns as listed below:

Another LatchProfX use case

Fri, 20 Mar 2009 19:35:41 +0000

Riyaj Shamsudeen wrote an excellent article about systematic latch contention troubleshooting. Especially if the latch contention problem is ongoing, looking into system wide stats (like v$latch.sleep columns) is not the best idea in busy systems. This may sometimes lead you to fixing the wrong problem. This is because sometimes the latch contention is not caused by some system wide inefficiency but rather by one or few sessions. The right approach would be to measure the following things:

The real history of Oracle database revealed!

Sat, 14 Mar 2009 05:36:40 +0000

Anyone who’s looked into Oracle X$ tables, knows that their names are really complicated and quite unreadable (and non-pronouncable), such X$KZSRT, X$KCPXPL, X$KQFSZ and so on. A few years ago at some conference someone came up with a thought that the reason why Oracle has so unreadable names for its X$ tables is that the leading edge database source code was actually stolen in the 80’s from a Soviet Union intelligence agency.

Oracle, Timesten and PL/SQL support

Sat, 07 Mar 2009 01:09:56 +0000

I thought to post about another new interest of mine, TimestTen, as I’ve worked with it in past and I have become a fan of it, especially after Oracle bought the company. Oracle has announced that TimesTen in-memory database will support PL/SQL in the upcoming release. That’s in 11gR2, where TimesTen is named the “in-memory database cache”. I’m happy to see the deep level of integration Oracle is doing with it.

The full power of Oracle’s diagnostic events, part 1: Syntax for KSD debug event handling

Wed, 04 Mar 2009 00:21:01 +0000

There’s a recent thread in Oracle-L about deadlocks and a recommendation to dump various instance information when the deadlock happens. A deadlock trace dumps some useful things automatically, but sometimes you want more, especially in RAC environment.

So is it possible to make Oracle dump additional things when the deadlock event happens? Yes it is and it’s doable with Oracle diagnostic event handling infrastructure.

First I’ll take a step back and explain, what this command below means:

SQL> alter session set events '10046 trace name context forever, level 12';

Session altered.

Of course you know what it does, it enables extended SQL trace. But why such cumbersome syntax?

This syntax actually reveals some of the power of KSD diagnostic event syntax (KSD=kernel service debug):

10046

The first word in event string (the 10046) specifies the when some action should be taken.

And everything after that (trace name context forever, level 12) is that action.

trace

The first word in action (trace) specifies what type of action Oracle needs to take if that event (10046) occurs. By “event occurs” I mean that let say an Oracle parsing function calls ksdpec() function with 10046 as parameter, which in turn may recursively call some action set for that event and then returns that event level (12) back to the caller. Its up to the caller to act on that returned value, in 10046 case some ksd* tracing function is called then.

In event syntax, “trace” is most generic action, which is used for tracing, behavior changing and Oracle dumps. In this post I will concentrate on the “trace” action as it’s most common one to use.

name

The “name” specifies that the name of what to dump/trace will follow. The “name” is always present when “trace” option is used (as far as I know).

context

Now the next keyword (context) is the one where you can define whether you want Oracle to dump something when the event is hit or just do context specific trace. If you replace the “context” with “errorstack” for example, you wouldn’t get SQL trace output, but rather an Oracle errorstack dump whenever event 10046 is hit.

You can use “oradebug dumplist” to find all the possible dump commands what you can set as actions for an event.

forever

The next keyword (forever) is actually an option to the action, not an action keyword itself. Forever means that keep invoking the action when the 10046 event is hit, forever (or until explicitly disabled). If we don’t specify forever, then the action would be invoked only once and the event will disable itself after that.

level 12

The “level 12” is also just another option to the action, specifying that the value for that event handler in given session should be 12. This means that whenever some Oracle function is checking whether that event is set, they will be returned value 12, the calling function interprets the value and acts appropriately (traces both binds and waits in our case).

As both the “forever” and “level 12” are just options for the same action, separated by commas, we can replace their order like that:

60000 bind variables?! Maybe it’s time to use a temporary table instead…

Wed, 25 Feb 2009 06:58:59 +0000

I just noticed a bug 8277300 filed in Metalink with following description: ORA-7445[XTYQBCB] OCCURS DURING EXECUTING SQL THAT USES 60000 BIND VARIABLES. Wow! That’s about 100 times more bind variables in a single query than what I’ve seen in past. And I thought that query was bad!!! :) I suspect this is a massive IN list passed to a query. Maybe it’s time to use a temporary table or a collection for passing in the IN values instead?

SQL_ID is just a fancy representation of hash value

Sun, 22 Feb 2009 21:02:12 +0000

…Or in other words, how to translate SQL_ID to a hash value :)

I once wrote a script to demo this in my Advanced Oracle Troubleshooting class.

Check this, I’ll run a query and then check what is its SQL_ID and HASH_VALUE from V$SQL:

SQL> select * from dual;

D
-
X

SQL> select sql_id, hash_value from v$sql
  2  where sql_text = 'select * from dual';

SQL_ID        HASH_VALUE
------------- ----------
a5ks9fhw2v9s1  942515969

So, V$SQL reports the real SQL_ID and HASH_VALUE above.

Now lets use my i2h.sql script ( i2h stands for: sql Id to Hash value ):

SQL> @i2h a5ks9fhw2v9s1

SQL_ID        HASH_VALUE
------------- ----------
a5ks9fhw2v9s1  942515969

It works! :)

The code itself is simple (and you can download the script from here)

Performance Visualization, Capacity planning and Hotsos Symposium

Sat, 14 Feb 2009 21:29:06 +0000

I have slowly moved towards performance visualization and system capacity planning world. Or at least this is my main focus (in addition to deep Oracle internals of course ;) I’ve published an easy Oracle performance visualization tool, which is based on Excel and can visualize the resultset of any SQL query. It’s called PerfSheet (and I’ve blogged about it here). Last December I spoke at CMG Conference in Las Vegas about a new method for visualizing database-wide performance while still being able to see session level performance profile.

When was a table last changed?

Sun, 08 Feb 2009 00:32:09 +0000

I frequently get a question about how to find out when was a table last modified. I’m talking about table data, not table structure, the latter would be detectable from dba_objects.last_ddl_time.

Unless you have some table level DML auditing already turned on, then as one option you could use LogMiner and “just” work through the redo/archivelogs in the range of interest with it. This could be very time consuming (especially if the last change was done a while back), so here’s another option for getting last table modification info very easily, however it comes with some restrictions.

https://github.com/tanelpoder/tpt-oracle/blob/master/lastchanged.sql

(read instructions and limitations from the script header).

The idea is following:

Oracle has an ORA_ROWSCN pseudocolumn which reports the last known change time for a row in a table. The “time” shows a commit SCN number of last transaction modifying the row, not a real timestamp though. It is important to note that unless the ROWDEPENDECIES are enabled, then the last SCN is known only at data block level, not row level, rowscn’s for all rows in a block would report whatever SCN is in the last change SCN in block header.
SCN is ever-increasing internal “time” used by Oracle recovery and transaction layers and it is possible to map this to real time with reasonable accuracy using few Oracle’s tables which store SCN to wallclock time mappings. My script reports the time range in which the last change to datablock/row occurred so you’ll know how accurate it is.

I use sys.smon\_scn\_time and v$log_history views for SCN to real time mapping. So my script reports two times, first one may be more accurate, second one has longer history on the other hand.You can use whatever other datasource for doing this mapping, as long as it has SCNs and corresponding timestamps in it. For example, if you have **_log\_checkpoints\_to_alert_** parameter set to true, you can grep the SCN/timestamp pairs out with command like this:</p> 
<pre>cat alert_win10g.log | egrep -e "^Beginning.*checkpoint|[[:alpha:]]{3} [[:alpha:]]{3} [[:digit:]]{2} "

Here are few usage examples.

First I’ll check some data which was probably last changed a long time ago:

SQL> @lastchanged sys.obj$ name='DBMS_STANDARD'

-- LastChanged.sql v1.0 by Tanel Poder (  )

Running this query:

.   select MAX(ora_rowscn)
.   from sys.obj$
.   where name='DBMS_STANDARD';

DATA_SOURCE       LAST_CHANGED
----------------- -------------------------------------------------------
sys.smon_scn_time Before  2008-12-31 16:05:25 (earlier than 21 days ago)
v$log_history     Before  2008-10-27 03:58:16 (earlier than 86 days ago)

2 rows analyzed.

As both sys.smon_scn_time and v$log_history don’t have records dating back to that old time when the database was created (when DBMS_STANDARD and other object records in that datablock were created), they just show that the change happened before the oldest SCN to time mapping they have.

Here’s a check on a regular table, but we are only interested in rows/datablocks where the “sal” column is bigger than 1000:

Multipart cursor subexecution and PRECOMPUTE_SUBQUERY hint

Fri, 23 Jan 2009 21:38:11 +0000

There was a question about PRECOMPUTE_SUBQUERY hint in an Oracle Forums thread.

Here I will post the answer I gave there and also elaborate it more as it explains a little known interesting fact about Oracle cursor management. Also it allows me to introduce few advanced Oracle troubleshooting scripts by example. This is a fairly long post, but if you are interested in some Oracle cursor management and SQL execution internals, keep on reading ;)

Consider the following test case with two tables, T1 and T2:

SQL> create table t1 as select rownum a from dual connect by level < 10;
Table created.
SQL> create table t2 as select rownum+10 b from dual connect by level < 10;
Table created.
SQL>
SQL> select * from t1;
     A


     1
     2
     3
     4
     5
     6
     7
     8
     9

9 rows selected.
SQL>
SQL> select * from t2;
     B


    11
    12
    13
    14
    15
    16
    17
    18
    19

9 rows selected.

Now lets run a query with a simple subquery in it:

SQL> select a
  2  from   t1
  3  where  a in (select b from t2);
no rows selected
SQL> select * from table(dbms_xplan.display_cursor(null,null,‘ALLSTATS LAST’));
PLAN_TABLE_OUTPUT
SQL_ID  aucw6byq3d5q8, child number 0
select a from   t1 where  a in (select b from t2)
Plan hash value: 561629455

| Id  | Operation           | Name     | E-Rows |  OMem |  1Mem | Used-Mem |
|   0 | SELECT STATEMENT    |          |        |       |       |          |
|*  1 |  HASH JOIN SEMI     |          |      1 |  1066K|  1066K| 1056K (0)|
|   2 |   TABLE ACCESS FULL | T1       |      9 |       |       |          |
|   3 |   VIEW              | VW_NSO_1 |      9 |       |       |          |
|   4 |    TABLE ACCESS FULL| T2       |      9 |       |       |          |
Predicate Information (identified by operation id):
1 - access(“A”=“B”)

Note that a hash semijoin was performed which semijoined two of its child rowsources with join condition “A=B”.

Now lets run exactly the same query with PRECOMPUTE_SUBQUERY hint in subquery block:

SQL> select a
  2  from   t1
  3  where  a in (select /*+ PRECOMPUTE_SUBQUERY */b from t2);
no rows selected
SQL>
SQL> select * from table(dbms_xplan.display_cursor(null,null,‘ALLSTATS LAST’));
PLAN_TABLE_OUTPUT
SQL_ID  fvnqhjkcjnybx, child number 0
select a from   t1 where  a in (select /*+ PRECOMPUTE_SUBQUERY */b from
t2)
Plan hash value: 3617692013

| Id  | Operation         | Name | E-Rows |
|   0 | SELECT STATEMENT  |      |        |
|*  1 |  TABLE ACCESS FULL| T1   |      5 |
Predicate Information (identified by operation id):
1 - filter((“A”=11 OR “A”=12 OR “A”=13 OR “A”=14 OR “A”=15 OR “A”=16
OR “A”=17 OR “A”=18 OR “A”=19))

See what happened! The join is gone and it looks like table T2 is not accessed at all (as there is only one TABLE ACCESS rowsource which reads from table T1 ).

However, there has appeared a filter condition which has all the values from T2 in it! How are these values retrieved?

A simple sql_trace reveals this:

Identifying shared memory segment users using lsof

Thu, 22 Jan 2009 12:42:45 +0000

Lsof (list open files) is a really useful tool for troubleshooting open file decriptors which prevent a deleted file from being released or a shared memory segment from being removed. Here’s a little situation on Linux where an Oracle shared memory segment was not released as someone was still using it. $ ipcs -ma ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 393216 oracle 640 289406976 1 dest 0xbfb94e30 425985 oracle 640 289406976 18 0x3cf13430 557058 oracle 660 423624704 22 ------ Semaphore Arrays -------- key semid owner perms nsems 0xe2260ff0 1409024 oracle 640 154 0x9df96b74 1671169 oracle 660 154 ------ Message Queues -------- key msqid owner perms used-bytes messages The bold line should have disappeared after instance shutdown, but it didn’t.

Reliable latch waits and a new blog

Tue, 20 Jan 2009 09:12:16 +0000

Here’s a link to Alex Fatkulin’s blog if you haven’t seen it already: http://afatkulin.blogspot.com/ He has some good Oracle internals information in there, I also like his research style. Alex just blogged about a finding (on Oracle 11g on Linux) that when Oracle process doesn’t get a latch after spinning, it goes to sleep using semop() system call, which never wakes up unless this semaphore is posted by another process. From past versions we remember that Oracle processes go to sleep for a short period of time, wake up, try to get the latch and sleep again for a longer period of time if unsuccessful (up to _max_exponential_sleep centiseconds).

Oracle memory troubleshooting, Part 1: Heapdump Analyzer

Fri, 02 Jan 2009 11:55:36 +0000

When troubleshooting Oracle process memory issues like ORA-4030’s or just excessive memory usage, you may want to get a detailed breakdown of PGA, UGA and Call heaps to see which component in there is the largest one.

The same goes for shared pool memory issues and ORA-4031’s – sometimes you need to dump the shared pool heap metadata for understanding what kind of allocations take most of space in there.

The heap dumping can be done using a HEAPDUMP event, see http://www.juliandyke.com/Diagnostics/Dumps/Dumps.php for syntax.

NB! Note that when dumping SGA heaps (like shared, large, java and streams pools), your process holds shared pool latches for the entire dump duration so this should be used only as a last resort in busy production instances. Dumping a big shared pool could hang your instance for quite some time. Dumping private process heaps is safer as that way only the target process is affected.

The heapdump output file structure is actually very simple, all you need to look at is the HEAP DUMP header to see in which heap the following chunks of memory belong (as there may be multiple heaps dumped into a single tracefile).

HEAP DUMP heap name="sga heap(1,1)"  desc=04EA22D0
 extent sz=0xfc4 alt=108 het=32767 rec=9 flg=-125 opc=0
 parent=00000000 owner=00000000 nex=00000000 xsz=0x400000
EXTENT 0 addr=20800000
  Chunk 20800038 sz=   374904    free      "               "
  Chunk 2085b8b0 sz=      540    recreate  "KGL handles    "  latch=00000000
  Chunk 2085bacc sz=      540    recreate  "KGL handles    "  latch=00000000
  Chunk 2085bce8 sz=     1036    freeable  "parameter table"
  Chunk 2085c0f4 sz=     1036    freeable  "parameter table"
  Chunk 2085c500 sz=     1036    freeable  "parameter table"
  Chunk 2085c90c sz=     1036    freeable  "parameter table"
  Chunk 2085cd18 sz=     1036    freeable  "parameter table"
  Chunk 2085d124 sz=      228    recreate  "KGL handles    "  latch=00000000
  Chunk 2085d208 sz=      228    recreate  "KGL handles    "  latch=00000000
  Chunk 2085d2ec sz=      228    recreate  "KGL handles    "  latch=00000000
  Chunk 2085d3d0 sz=      228    recreate  "KGL handles    "  latch=00000000
  Chunk 2085d4b4 sz=      228    recreate  "KGL handles    "  latch=00000000
  Chunk 2085d598 sz=      540    recreate  "KQR PO         "  latch=2734AA00
  Chunk 2085d7b4 sz=      540    recreate  "KQR PO         "  latch=2734AA00
  Chunk 2085d9d0 sz=      228    recreate  "KGL handles    "  latch=00000000
...

The first list of chunks after HEAP DUMP (the list above) is the list of all chunks in the heap. There are more lists such as freelists and LRU lists in a regular heap, but lets ignore those for now, I’ll write more about heaps in an upcoming post.

After identifying heap name from HEAP DUMP line, you can see all individual chunks from the “Chunk” lines. The second column after Chunk shows the start address of a chunk, sz= means chunk size, the next column shows the type of a chunk (free, freeable, recreate, perm, R-free, R-freeable).

The next column is important one for troublehsooting, it shows the reason why a chunk was allocated (such KGL handles for library cache handles, KGR PO for dictionary cache parent objects etc). Every chunk in a heap has a fixed 16 byte area in the chunk header which stores the allocation reason (comment) of a chunk. Whenever a client layer (calling a kghal* chunk allocation function) allocates heap memory, it needs to pass in a comment up to 16 bytes and it’s stored in the newly allocated chunk header.

This is a trivial technique for troubleshooting memory leaks and other memory allocation problems. When having memory issues you can just dump all the heap’s chunks sizes and aggregate these by allocation reason/comment. That would show you the biggest heap occupier and give further hints where to look next.

As there can be lots of chunks in large heaps, aggregating the data manually would be time consuming (and boring). Here’s a little shell script which can summarize Oracle heapdump output tracefile contents for you:

Performance Visualization made easy – PerfSheet 2.0 beta

Sun, 28 Dec 2008 15:19:36 +0000

Update: Luca Canali has written a newer version of Perfsheet (PerfSheet v4) and you can download it here. I am not updating/maintaining the old versions anymore (and Perfsheet v3 is only tested on up to Excel 2007) so I’d recommend to check out Luca’s PerfSheet v4.

Hi all,

I have been extremely busy over last couple of months, that’s why there haven’t been any blog entries (no, I haven’t ran out of good ideas ;-)

I just managed to find some time on a day-time 9-hour flight from Shanghai to Finland, thus here’s a blog entry about something what I had wanted to write about for long time… I’m talking about the PerfSheet tool I wrote a year ago and have been showing at few conferences already. It has been a great time saver for me over this year when working through performance data for troubleshooting or capacity planning tasks.

If you don’t want to read further through my comments, just download PerfSheet from </files/PerfSheet.zip> and see how it works yourself!

For others, let me give some history first:

A little oradebug enhancement in Oracle 11g

Sun, 02 Nov 2008 18:47:27 +0000

There’s a command called: ORADEBUG SETORAPNAME in 11g. It allows you to attach to a named background process as shown below, so you don’t need to figure out what’s the PID or SPID of the target process. SQL> oradebug setorapname dbw0 Oracle pid: 9, Unix process pid: 5506, image: oracle@linux03 (DBW0) SQL> SQL> oradebug setorapname pmon Oracle pid: 2, Unix process pid: 5490, image: oracle@linux03 (PMON) SQL> Even though you probably don’t want to mess around with background processes in production DBs, in demos and just Oracle research it can help you save couple of seconds every now and then.

Advanced Oracle Troubleshooting Guide, Part 9 – Process stack profiling from sqlplus using OStackProf

Fri, 31 Oct 2008 01:02:17 +0000

Update: I wrote a Python-based OStackProf stack trace summarizer that works on MacOS and Linux/Unix clients too (possibly even on Windows client if you have Python installed).

I have mentioned ORADEBUG SHORT_STACK command in my blog posts before – it’s an easy way to get and see target processes stack backtrace directly in sqlplus. No need to log on to the Unix/Windows server and use OS tools for extracting the stack.

I have also written few tools which allow you to post-process stack traces taken using OS tools (like pstack) for better readability or performance profiling. For example os_explain and DStackProf – the DTrace stack profiler and function call aggregator.

Now I will introduce OStackProf which combines the ORADEBUG SHORT_STACK with a client side post-processing script for easy stack profiling directly from SQLPLUS – no need to log on to the server host at all!

I have demonstrated this script at conferences for couple of months now with a promise to blog about it “soon”, but it’s only now that I finally have a chance to write a blog entry about it (I have couple of hours left on a flight to help a client in London). Sorry to keep you waiting ;-)

So, this is what you see when you run oradebug short_stack on a process:

SQL> oradebug setospid 32200
Oracle pid: 2, Unix process pid: 32200, image: oracle@linux03 (PMON)
SQL>
SQL> oradebug short_stack
<-ksedsts()+275<-ksdxfstk()+22<-ksdxcb()+1599<-sspuser()+102<-__kernel_vsyscall()+2<-ntevpque()+89<-ntevqone()+34<-nsevwait()+10098<-ksnwait()+72<-ksliwat()+7249<-kslwaitctx()+135<-ksuclnwt()+249<-ksucln()+509<-ksbrdp()+1258<-opirip()+548<-opidrv()+500<-sou2o()+71<-opimai_real()+238<-ssthrdmain()+142<-main()+116<-__libc_start_main()+220<-_start()+33
SQL>

This is a stack trace, the leftmost function is where the execution currently was at the moment of taking the stack backtrace, the one to the right from it is the caller of the left function and so on. All the way to the right you see the “bottom” functions of a stack, like C main() function and also _start() which is Linux OS process loader helper function.

Note the two bold functions. Oradebug short_stack works the way that whenever it needs to let target process know about a debugger request, it sends a SIGUSR2 signal to it. sspuser() is the signal handler for SIGUSR2 and its task is to see what debug function to call. So, the sspuser() function and anything to the left from it is actually the codepath for processing the oradebug request, so we can ignore that for troubleshooting purposes (this also evidences that oradebug short_stack and dump errorstack as matter of fact do stray Oracle from its normal execution path – OS tools don’t do that, they just suspend the process and read what’s needed from process stack frames and mapped executable symbol sections).

So, the real “business” function the target was in, was __kernel_vsyscall() which in Linux means the process was doing some sort of syscall. I’m not going further in explaining the interpretation of function names here as I’ve done it in my previous AOT posts (and will continue so in the future).

I will continue on stack profiling topics. Taking only single stack backtrace can be helpful in cases when the target process is completely stuck, but if you want to diagnose the cause for just bad performance, then you need to take multiple stack backtraces, aggregate them and see into which execution “branch” of the codepath do the most stack samples fall in (as DStackProf does for example). Of course you would do this only after you’ve exhausted the step 1 and 2 in normal Oracle troubleshooting process (1. Look into wait interface data for the session, 2. Look into v$sesstat counters for the session).

Manually aggregating the stack traces is time consuming, error prone and did I mention boring, so now I finally introduce OStackProf v1.00!

Transportable tablespaces and ROWID uniqueness

Tue, 21 Oct 2008 01:25:29 +0000

I recently saw a fellow OakTable member mentioning a section in Oracle documentation where it’s said that:

“When a database contains tablespaces that have been plugged in (from other databases), the ROWIDs in that database are no longer unique. A ROWID is guaranteed unique only within a table.”

It’s a well known fact that the old Oracle7 style restricted rowids (which contained only File#, block# and row#) may not be unique in Oracle8+ databases which can have 1022 datafiles per tablespace not per database as previously. That’s why the 10-byte extended rowids were introduced, which also included the data object ID of a segment inside the rowid.

So if you have a global (partitioned) index on a partitioned table, the new extended rowids are used in it. With help of data dictionary cache this allows Oracle to quickly figure out in which tablespace the referenced segment resides and then use old fashioned relative-file#, block# and row# lookup on it.

Note that with local indexes and non-partitioned tables the rowids stored in indexes are old 6-byte restricted rowids. They have 4 bytes for data block address consisting of 10bits for file# and 22bits for block#. The other 2 bytes specify the row# in block.

The 10-byte extended rowids used to be unique within a database, until the transportable tablespaces came into play. Why is that – it’s because how transportable tablespaces work.

The idea behind TTS’es is that one should be able to copy (large) tablespaces around at file level, very fast, without much pre- or post-processing. So when we copy an 1TB TTS around, we don’t want to start scanning through it after plug-in to make ROWIDs in indexes and chained row forward pointers to somehow “make them right”. Thus, when plugging in a TTS, we need to keep all low-level row addressing structures untouched. This means that file#, block# and row# parts in index leaves and chained row pointers need to stay as they are. This means that DBAs (data block addresses in ASSM blocks need to stay as they are).

Also, as every segment data block holds a data object ID in it (which is also part of extended rowid as I mentioned), we can’t change this either, as otherwise we would need to scan through all tables/indexes in that tablespace again.

Thus, if all components of extended ROWID of an imported TTS need to stay as they are, there is theoretical and practical chance of ROWID collision within a database (note that I said within a database, not table).

A test case for demonstrating this is quite simple:

VLDB 2008 proceedings, Oracle optimizer plan stability, adaptive cursor sharing and SecureFiles

Fri, 05 Sep 2008 12:11:45 +0000

If you’re interested in leading edge database research (as of 2008 :), the VLDB 2008 proceedings are publicly available now. Here are direct links to some Oracle-specific ones: Oracle SecureFiles System Optimizer Plan Change Management: Improved Stability and Performance in Oracle 11g Closing The Query Processing Loop in Oracle 11g (this one is about adaptive cursor sharing) Enjoy! :)

Oracle hidden costs revealed, Part2 – Using DTrace to find why writes in SYSTEM tablespace are slower than in others

Mon, 01 Sep 2008 17:16:50 +0000

_I have written two posts in one, about a performance issue with writes in system tablespace and introduction of a little DTrace stack sampling script.

Have you noticed that DML on tables residing in SYSTEM tablespace is slower than tables in other tablespaces?

Here’s an example, I’ll create two similar tables, one in USERS tablespace, other in SYSTEM, and inset into the first one (Oracle 10.2.0.3 on Solaris x64):

SQL> create table t1(a int) tablespace USERS;

Table created.

SQL> create table t2(a int) tablespace SYSTEM;

Table created.

SQL> exec for i in 1..100000 loop insert into t1 values(i); end loop;

PL/SQL procedure successfully completed.

Elapsed: 00:00:03.09

Insert into table in USERS tablespace took 3 seconds.

Ok, let’s commit and flush dirty buffers that they wouldn’t get on the way of next insert.

SQL> commit;

Commit complete.

Elapsed: 00:00:00.02

SQL> alter system checkpoint; -- checkpointing to flush dirty buffers from previous inserts

System altered.

Elapsed: 00:00:01.34
SQL>

And now to the insert into the SYSTEM tablespace table:

SQL> exec for i in 1..100000 loop insert into t2 values(i); end loop;

PL/SQL procedure successfully completed.

Elapsed: 00:00:08.98
SQL>

What?! The same insert took 3 times longer, almost 9 seconds?

Fine! Let’s troubleshoot it!

Flexible sampling of any V$ or X$ view with sample.sql

Tue, 26 Aug 2008 10:35:00 +0000

In recent past I’ve blogged few scripts which use specially crafted ordered nested loop for sampling contents of V$ and X$ views fast, with plain SQL.

If you haven’t read them yet, here are the links:

I wrote the above scripts having special purposes in mind (e.g. profile session waits or latching activity).

Now I introduce a simple but powerful sqlplus script for ad-hoc sampling of any V$ view. It’s called…. (drumroll) …. sample.sql :)

When you look into it, the script is actually very simple. It’s just using power of sqlplus substitution variables, I can pass the sampled column and table names and sampling conditions in to the script dynamically.

The basic syntax is:

@sample column_name table_name filter_condition num_samples

For example, let say I have a session which executes lots of SQL statements in a loop and I want to have a quick overview of what’s the TOP SQL statement for a session right now.

Case study on some rowcache internals, cached non-existent objects and a describe bug

Mon, 18 Aug 2008 12:37:20 +0000

I got a question regarding Metalink note 296235.1 about a describe bug which causes objects to “disappear” when they are described when database is not open. It was an interesting case involving a bug, so I wrote a quite long analysis with test cases today. However when posting the entry to wordpress, it managed to completely mess up the formatting. After wasting half an hour trying to get the formatting correct I gave up and saved the article into a PDF instead.

Why does even a small difference in SQL text cause a hard parse?

Thu, 14 Aug 2008 18:45:52 +0000

I just replied to an Oracle Forum Thread about why does even a small difference in SQL statement text cause it to be hard parsed and loaded as a different cursor. The reason is actually very simple – and I’m posting it into my blog too: — The reason why a statement with even a minor difference in text is parsed as a separate cursor is due how Oracle looks up statements from library cache.

Script: Display valid values for multioption parameters (including hidden parameters)

Tue, 12 Aug 2008 18:16:58 +0000

I wrote a little script pvalid.sql for listing valid values for multioption parameters (the ones which are not string, number or boolean type, but accept a parameter from predetermined list, like optimizer_mode which can have values of ALL_ROWS, FIRST_ROWS, CHOOSE, FIRST_ROWS_1, etc).

The script accepts a (part of) Oracle parameter name as first argument, for example the following output is from Oracle 10.2.0.3 database:

SQL> @pvalid lock

  PAR# PARAMETER                                          VALUE                          DEFAULT
------ -------------------------------------------------- ------------------------------ -------
   374 _db_block_cache_protect                            FALSE                          DEFAULT
       _db_block_cache_protect                            LOW
       _db_block_cache_protect                            MEDIUM
       _db_block_cache_protect                            TRUE

   376 db_block_checksum                                  TRUE                           DEFAULT
       db_block_checksum                                  FALSE
       db_block_checksum                                  FULL
       db_block_checksum                                  OFF
       db_block_checksum                                  TYPICAL

   696 _row_locking                                       ALWAYS
       _row_locking                                       ALWAYS
       _row_locking                                       DEFAULT
       _row_locking                                       DEFAULT
       _row_locking                                       INTENT
       _row_locking                                       INTENT

   756 db_block_checking                                  FALSE                          DEFAULT
       db_block_checking                                  FULL
       db_block_checking                                  LOW
       db_block_checking                                  MEDIUM
       db_block_checking                                  OFF
       db_block_checking                                  TRUE

   851 _plsql_anon_block_code_type                        INTERPRETED                    DEFAULT
       _plsql_anon_block_code_type                        NATIVE


23 rows selected.

SQL>

Or:

The simplest query for checking what’s happening in a database

Wed, 06 Aug 2008 16:30:43 +0000

Update: If you are able to download & run scripts, it’s probably better to use my Snapper and @ashtop.sql tools for performance analysis. Neither of these scripts require any installation, Ashtop requires Diagnostics Pack (as it uses ASH), Snapper doesn’t even need that - it can also run on Standard Edition! Search for them in my blog, or check an intro video in my YouTube channel. When someone asks you to take a quick look into database performance and for whatever reason you can’t run your usual scripts or performance tools on there, ), then what query would you run first?

Library cache latches gone in Oracle 11g

Sat, 02 Aug 2008 19:54:34 +0000

In Oracle 11g even more library cache operations have been changed to use KGX mutexes instead of latches.

In Oracle 10.2.0.2+ the library cache pin latch usage was replaced with mutexes whenever _kks_use_mutex_pin was true, also few other things like V$SQLSTATS arrays and parent cursor examination were protected by mutexes. However the traversing of library cache hash chains (the right child cursor lookup using kksfbc()) was still protected by library cache latches which could become a problem with frequent soft parsing combined with too little cursor cache and long library cache hash chains (remember, the library cache latches were always taken exclusively even for plain hash chain scanning).

In 11g all library cache related latches except “library cache load lock” are gone and corresponding operations are protected by mutexes instead. The “library cache” latches have been replaced by “Library Cache” mutexes for example.

Here are couple queries which illustrate the change.

**Executed on 10.2.0.3:

SQL> select name from v$latch where lower(name) like '%library%';

NAME
--------------------------------------------------
library cache pin allocation
library cache lock allocation
library cache hash chains
library cache lock
library cache
library cache pin
library cache load lock

7 rows selected.

SQL> select name from v$event_name where name like '%library%';

NAME
----------------------------------------------------------------
latch: library cache
latch: library cache lock
latch: library cache pin
library cache pin
library cache lock
library cache load lock
library cache revalidation
library cache shutdown

8 rows selected.

Same queries executed on 11.1.0.6 and the bold lines above are gone:

Advanced Oracle Troubleshooting Guide, Part 8: Even more detailed latch troubleshooting using LatchProfX

Tue, 22 Jul 2008 16:58:23 +0000

In my last AOT post I published my LatchProf script which is able to sample detailed latchholder data from V$LATCHHOLDER.

Latchprof allows you to drill down into your latching problems at session level (which V$LATCH, V$LATCH_PARENT and V$LATCH_CHILDREN can’t do). It allows you to get valuable details about individual sessions who are holding a latch the most, therefore likely contributing to the latch contention problem the most.

However after you have discovered the troublemaking session, then what next? One way forward is looking into V$SESSTAT counters using Snapper tool. Depending on what latch is the problematic one, you would look for different stats like various buffer get stats for cache buffers chains latches and parsing/executing stats when looking into library cache latches. However if those stats look “normal”, is there any other way do drill down further?

Yeah, there is and lets look into it!

Closed database and WITH subquery

Mon, 14 Jul 2008 11:17:07 +0000

Here’s an interesting issue I found when running a query using WITH subquery factoring when database was not open (it was in NOMOUNT mode in current case). As you probably know you can query DUAL table when database is not open, but in this case the actual query is made against X$DUAL as seen below: SQL> select * from dual; ADDR INDX INST_ID DUM -------- ---------- ---------- --- 051ED14C 0 1 X SQL> When you have above fields when querying from DUAL then you know your database is probably not open.

Advanced Oracle Troubleshooting Guide, Part 7: Sampling latch holder statistics using LatchProf

Tue, 08 Jul 2008 11:54:10 +0000

I have been too busy since getting back from vacation, thus no posts for a while. But I hope the waiting was worthwhile as I present you LatchProf, a tool for digging in to latch contention problems – using plain SQL and sqlplus!

As, I’m still busy, I make it short.

LatchProf is a script similar to WaitProf, only it samples latch holder statistics from V$LATCHHOLDER. As V$LATCHHOLDER contains a SID column (with session ID of a latch holder) it becomes possible to find who is hitting a latch the most (a way to prove that crappy monitoring tools which constantly scan through V$SQL DO cause library cache latch contention themselves).

Another use case for WaitProf – diagnosing “events in waitclass Other”

Fri, 20 Jun 2008 16:46:47 +0000

I recently diagnosed a performance issue where the “events in waitclass Other” occasionally took significant part of the session’s response time. For example Snapper (which reads wait event data from V$SESSION_EVENT) reported that during measuring 39.9% of the response time was spent on “events in waitclass Other”.

SQL> @sn 1 119

-- Session Snapper v1.07 by Tanel Poder (  )

---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,  DELTA/SEC,     HDELTA, HDELTA/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     119, 20080621 05:22:05,        1, STAT, session logical reads                   ,         18284,      18284,     18.28k,     18.28k
DATA,     119, 20080621 05:22:05,        1, STAT, consistent gets                         ,         15301,      15301,      15.3k,      15.3k
DATA,     119, 20080621 05:22:05,        1, STAT, consistent gets from cache              ,         15228,      15228,     15.23k,     15.23k
DATA,     119, 20080621 05:22:05,        1, STAT, consistent gets from cache (fastpath)   ,         15136,      15136,     15.14k,     15.14k
DATA,     119, 20080621 05:22:05,        1, STAT, calls to get snapshot scn: kcmgss       ,            89,         89,         89,         89
DATA,     119, 20080621 05:22:05,        1, STAT, no work - consistent read gets          ,         14883,      14883,     14.88k,     14.88k
DATA,     119, 20080621 05:22:05,        1, STAT, table scans (short tables)              ,            21,         21,         21,         21
DATA,     119, 20080621 05:22:05,        1, STAT, table scan rows gotten                  ,       1429227,    1429227,      1.43M,      1.43M
DATA,     119, 20080621 05:22:05,        1, STAT, table scan blocks gotten                ,         17440,      17440,     17.44k,     17.44k
DATA,     119, 20080621 05:22:05,        1, WAIT, events in waitclass Other               ,        399831,     399831,   399.83ms,   399.83ms
--  End of snap 1

From Oracle 10g Oracle has consolidated lots of the events into “events in waitclass Other”. This is because saving all 900+ wait event stats for every session (in V$SESSION_EVENT array) would waste too much memory with giving little benefit (normally there’s only a handful of troublemaking events anyway). Therefore makes sense to aggregate the least likely happening events under some common category. Looks like Oracle kernel coders have set a threshold in event number above which all events are grouped under the “other” waitclass.

See below, this is from 11g:

Killing an Oracle process from inside Oracle

Thu, 19 Jun 2008 11:44:10 +0000

I had a following situation few days ago – I was running a CREATE TABLE AS SELECT over a heterogenous services dblink. However I cancelled this command via pressing CTRL+C twice in Windows sqlplus (this actually just kills the client sqlplus and not the call).

Anyway, when I wanted to drop that table involved, this happened:

SQL> drop table MYTABLE;
drop table MYTABLE
           *
ERROR at line 1:
ORA-00054: resource busy and acquire with NOWAIT specified

I can’t drop a table as someone is holding a lock on it. Fair enough, this was a dev environment used only by me, so I used DBA_OBJECTS.OBJECT_ID to find out the object ID of that table:

SQL> @o MYTABLE

owner                     object_name                    object_type        CREATED           LAST_DDL_TIME     status           OID      D_OID
------------------------- ------------------------------ ------------------ ----------------- ----------------- --------- ---------- ----------
XYZ_DEV01_OWNER           MYTABLE                        TABLE              20080616 11:08:44 20080616 11:08:44 VALID          63764      63764

…and then I queried what enqueue locks were held on that object:

SQL> select * from v$lock where id1=63764;

ADDR     KADDR           SID TY        ID1        ID2      LMODE    REQUEST      CTIME      BLOCK
-------- -------- ---------- -- ---------- ---------- ---------- ---------- ---------- ----------
40034278 40034290        130 TM      63764          0          6          0       2662          0

Ok, I see session 130 holding a TM lock on that table. I queried the corresponding SERIAL# from v$session as well and killed the session:

SQL> alter system kill session '130,8764';
alter system kill session '130,8764'
*
ERROR at line 1:
ORA-00031: session marked for kill

SQL> select * from v$lock where id1=63764;

ADDR     KADDR           SID TY        ID1        ID2      LMODE    REQUEST      CTIME      BLOCK
-------- -------- ---------- -- ---------- ---------- ---------- ---------- ---------- ----------
40034278 40034290        130 TM      63764          0          6          0       2668          0

After hanging for 60 seconds, my kill command gave up (and marked my session for kill), but my lock was still not released… Now what?

cursor_space_for_time To Be Deprecated

Tue, 17 Jun 2008 10:52:55 +0000

If you haven’t seen the Meatlink note 565424.1 in the news yet, cursor_space_for_time parameter will be deprecated in Oracle 10.2.0.5 and 11.1.0.7. That’s kind of good news, I hope this will eventually reduce the number of expert DBAs who set this parameter to true whenever they see any kind of shared pool / library cache latch contention. On the other hand, spin_count was made an undocumented parameter long time ago, but is still heavily abused worldwide so I wouldn’t be surprised if the same happens to future _cursor_space_for_time…

Advanced Oracle Troubleshooting Guide, Part 6: Understanding Oracle execution plans with os_explain

Sun, 15 Jun 2008 14:40:12 +0000

Get ready for some more adventures in Oracle process stack!

Before proceeding though, please read this post about safety of different stack sampling approaches.

I have had few non-trivial Oracle troubleshooting cases, related to query hangs and bad performance, where I’ve wanted to know where exactly in execution plan the current execution is.

Remember, Oracle is just another program executing instructions clustered in functions on your server, so stack sampling can help out here as well.

So, I was looking into the following stack trace taken from an Oracle 10.1 database on Solaris SPARC, running a SQL with this execution plan.

Debugger dangers

Sat, 14 Jun 2008 11:51:54 +0000

Whenever I deliver training or conference presentations on advanced troubleshooting topics, I usually spend some time demonstrating how to get and interpret Oracle server process stack traces.

As I’ve mentioned before, stack traces are the ultimate indicators showing where in Oracle kernel (or whatever application) code the execution currently is (or where it was when a crash occurred). This is the reason Oracle Support asks for stack traces whenever there’s a crash or non-trivial hang involved, that’s why Oracle database dumps errorstacks when ORA-600’s and other exceptions occur.

There are multiple ways for getting stack traces for Oracle, but not all ways are equal. Some give you more contextual info, some less, but what I’m blogging about today is that some ways are less safe than others.

I was using pstack on Linux for diagnosing an IO related performance issue. I executed a create table as select statement and ran pstack in a loop for getting stack traces from the running process.

However in one of the test runs I got following error in my Oracle session:

SQL> create table t as select * from dba_source;
create table t as select * from dba_source
                                *
ERROR at line 1:
ORA-01115: IO error reading block from file 1 (block # 11161)
ORA-01110: data file 1: '/u01/oradata/LIN10G/system01.dbf'
ORA-27091: unable to queue I/O
ORA-27072: File I/O error
Additional information: 3
Additional information: 11145
Additional information: 32768

I suspected that this issue was due Linux pstack, stopped the pstack script and ran my CTAS from the same Oracle session again:

Short note on KGX Mutexes

Thu, 12 Jun 2008 15:01:14 +0000

I received a question on what’s the point of the use of Mutexes for Oracle cursors in library cache. For short intro, I’m pasting one of my fairly recent answers in Oracle forums about Oracle mutexes here: In Oracle, latches and mutexes are different things and managed using different modules. KSL* modules for latches and KGX* for mutexes. General mutex operatins require less CPU instructions than latch operations (as they aren’t as sophisticated as latches and don’t maintain get/miss counts as latches do).

Generating lots of rows using connect by – safely!

Sun, 08 Jun 2008 09:06:20 +0000

Every now and then I need to generate a large number of rows into some table.

Not that I have any personal urge to do so, but you know, this is needed for some test cases and so on ;-)

It’s quite well known that it’s possible to generate lots of rows using CONNECT BY LEVEL <= #rows syntax.

However as the CONNECT BY LEVEL syntax is recursive by nature, it will consume more memory the more rows you query in one shot ( I think it was Mark Bobak who once posted this observation to Oracle-L, but I didn’t manage to find that thread anymore ).

So, here’s a test case:

SQL> select count(r)
  2  from (
  3     select rownum r from dual connect by rownum <= 100000000
  4  )
  5  /
        select rownum r from dual connect by rownum <= 100000000
                             *
ERROR at line 3:
ORA-04030: out of process memory when trying to allocate 44 bytes (kxs-heap-w,cursor work heap)

After running for a while the server process run out of private memory, used for the CONNECT BY cursor work heap.

Let’s investigate:

Advanced Oracle Troubleshooting Guide, Part 5: Sampling V$ stuff with WaitProf. Really fast. Using SQL!

Thu, 05 Jun 2008 17:11:42 +0000

I bet you thought I’ll be writing about direct SGA access?! ;)

Nope!

Direct SGA access has excellent troubleshooting potential (as long as you know the shared memory data structures), but it has one major drawback – very few companies have such tools already in place in their production systems.

I have occasionally been called in to solve an urgent performance problem, happening right now and it needs solving immediately! And did I mention, these are critical production systems. Where you can’t just install binary executables freshly downloaded off internet. In fact you would want to diagnose the issue with minimal impact and changes required to those production environments (and that leaves sql tracing out the first round troubleshooting tools for me as well!)

So, I’ve developed myself a toolset for such purpose, Snapper and sw.sql and some process stack reading techniques I already have introduced in my blog.

Next in line is waitprof.sql which is a high-frequency V$SESSION_WAIT sampler – implemented in plain SQL (not PL/SQL).

Waitprof is basically a sampling session wait profiler. It’s like running a select against V$SESSION_WAIT in a very tight loop and aggregating results – but I have used a trick to do all this in plain SQL, which gives me performance advantage over PL/SQL based loops. Waitprof is able to sample V$SESSION_WAIT for a session up to 100 000 times per second!

This depends on your hardware of course and Oracle version too, but normally you’ll get 50-70kHz sampling rate with it.

Ok, you want to see an example? ;-)

Performance Tools Quick Reference Guide

Mon, 02 Jun 2008 18:09:23 +0000

There’s a nice Metalink Note 438452.1 about various less known Oracle performance tuning utilities. If you haven’t heard about things like StackX, LTOM, HangFG, SQLTXPLAIN, OS_Watcher or OPDG then it’s time to check this note out! :)

Advanced Oracle Troubleshooting Guide, Part 4: Diagnosing a long parsing issue

Mon, 02 Jun 2008 18:00:38 +0000

There was a recent thread in Oracle Forums about a session getting stuck somewhere when a specific SQL was issued. The SQL executed did not return at all unless ORDERED hint was used. Even the EXPLAIN PLAN command (which only parses the statement, doesn’t execute it) did never return.

Classic tracing + tkprof techniques didn’t show much (just some recursive queries consuming insignificant amounts of time).

The proven V$SESSION_WAIT sampling technique didn’t reveal anything as it showed the session being constantly on CPU (the wait state = ‘WAITED KNOWN TIME’ which means session is on CPU) and SEQ# didn’t increase (which means that wait state did not change over time).

Due the symptoms described above I was well prepared to troubleshoot this issue. This looks exactly like one of the troubleshooting use cases I demonstrate in of my Advanced Oracle Troubleshooting class (nice embedded advertisment, huh? ;)

In such a case where tracing and V$ views don’t provide any useful information about what the session is doing, I normally look into few stack traces of the server process. In this case I asked the poster to do this and here is the result:

Snapper shortcut

Mon, 02 Jun 2008 15:05:02 +0000

I have a (very) small script called sn.sql which I use as a wrapper around snapper (maybe I should’ve called it Snapper Wrapper but it’s too long name for the purpose :) The idea is to have to type less when running Snapper with default options (take 1 snapshot, output to screen and display Session tats,Wait events and Time model stats). Whenever there’s a performance issue with a session I first quickly run @sn , for example:

Querying the current tracefile name, using SQL – with tracefile_identifier

Fri, 30 May 2008 17:41:22 +0000

Update: In Oracle 11g upwards you can use V$DIAG_INFO for getting your own session’s trace file name too (diag.sql script). But on earlier versions or if you want to see the current tracefile name of another, you still need to use the V$PROCESS approach below. Note that V$PROCESS in 11g (or 11.2?) has a column TRACEFILE that shows the entire path+name of a process tracefile (however there seems to be a bug in 12.

Oracle Troubleshooting with Snapper – detecting who’s causing excessive redo generation

Fri, 30 May 2008 14:13:38 +0000

Update: As this post was written many years ago, you should check out newer Snapper articles/videos:

/2013/02/18/snapper-v4-02-and-the-snapper-launch-party-video/

My friend asked today a question that how to identify why his Oracle 9.2 database has suddenly started generating loads more redo than usual.

So obviously I recommended him Snapper as first thing, it’s perfect for ad-hoc analysis like that! ( I know I sound biased but if you haven’t used Snapper yet, then now is the time! :)

So, I asked him to run Snapper on all sessions of the instance with 10 second interval and find the session with highest “redo size” delta figure.

However as a normal production database has loads of sessions in it and as each session can return tens of statistic rows, then you might need to do some post-processing of output data to get only the “redo size” numbers.

So I figured that why not do the filtering in Snapper itself, so I made the small additions and released Snapper v1.07.

Usage is simple:

SQL> @snapper out,gather=s,sinclude=%redo_size% 10 1 "select sid from v$session"

-- Session Snapper v1.07 by Tanel Poder (  )

---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,  DELTA/SEC,     HDELTA, HDELTA/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     129, 20080530 09:07:20,       10, STAT, redo size                               ,     180595100,   18059510,     180.6M,     18.06M
DATA,     166, 20080530 09:07:20,       10, STAT, redo size                               ,        790912,      79091,    790.91k,     79.09k
DATA,     167, 20080530 09:07:20,       10, STAT, redo size                               ,        889796,      88980,     889.8k,     88.98k
--  End of snap 1

PL/SQL procedure successfully completed.

And in the result you already see that there is a single session which generates significantly more redo than others (btw this is an artificially generated example). In my friend’s case there also happened to be a single session , scheduled through DBMS_JOB, contributing to most of the redo generation. Once the troublemaker was identified, the problem could be fixed.

There are few things to note:

Performance and Scalalability Improvements in Oracle 10g and 11g

Sun, 25 May 2008 12:08:27 +0000

I have uploaded the slides of my “Performance and Scalalability Improvements in Oracle 10g and 11g” presentation here.

Using autonomous transactions for sleeping

Sun, 25 May 2008 09:52:45 +0000

There was a question in a recent Oracle-L thread about various uses of autonomous transactions.

Autonomous transactions can be very useful for a PL/SQL application logging, but sometimes they are also abused to cope with bad application design (like avoiding mutating table errors in triggers etc).

I’m not going to start on that topic here though, but instead presenting another case where autonomous transactions have helped me to work around a problem. It’s more a hack than a real solution though, but may be useful for someone else too.

My Snapper tool requires execute rights on DBMS_LOCK in order to sleep between snapshots. Sometimes when troubleshooting an urgent performance issue, I have had access only to some kind of application support account, without permissions to execute DBMS_LOCK.SLEEP. And sometimes it takes too long to get those rights granted by corporate DBAs.

So one workaround I’ve used so far is creating a fake DBMS_LOCK.SLEEP proc in the local support schema along with one dummy table and use a combination of SELECT FOR UPDATE and autonomous transactions to sleep for short time.

The code is very simple:

SQL*Net break/reset to client

Wed, 09 Apr 2008 16:50:49 +0000

There was a question in Oracle-L mailinglist recently, regarding excessive SQL*Net break/reset to client waiting by a session.

A quote is below:

We are seeing an insert statement reporting SQL*Net break/reset to client as over 33% of its time.

On the face of it this event suggests network issues but nothing else backs this up as the cause.

So I looked at the Java code in question and a trace of one of the sessions.

What is happening is that an attempt is made to insert a row, most of the time a duplicate error results, the code catches this exception and does an update.

I was wondering if its the duplicate error and the exception handling which results in this wait event showing up.

My answer to that was following:

Yes, a SQL*Net break/reset happens when an error/unhandled exception is raised during a call (which means that the call executed didn’t complete normally, thus the call state must be reset).

The approach of “insert -> if failed then update” is basically what MERGE does.

You need to change the application to use MERGE command. Alternatively you could use an “update -> if no-rows-updated then insert” approach, but MERGE makes much more sense nowadays.

So, these waits aren’t really indicating any network bottleneck, but rather just the fact that the client needs to be notified if it’s call has failed and Oracle has to clean up after the failed call. Apparently on some (network protocol) architectures or in some scenarios these operations have taken significant time (and may have required a system call), so a kernel developer has decided to wait instrument them.

In this blog entry I want to share a small test case which illustrates the point. First lets check how many times my session has waited for a SQL*Net break/reset wait event:

SQL> select event, total_waits from v$session_event where event like '%reset%' and sid = (select sid from v$mystat where rownum = 1);

no rows selected

As V$SESSION_EVENT (like also V$SYSTEM_EVENT) doesn’t display events for which nobody has waited yet, I see that my session’s wait count for that event is zero.

Now let’s create an error, but also let’s handle it in a PL/SQL exception handler:

SQLNet message to client vs SQLNet more data to client

Sun, 10 Feb 2008 07:35:58 +0000

After my last post about SQL*Net message to client wait event I had a follow-up question about what’s the difference between SQL*Net message to client and SQL*Net more data to client wait events. I’ll post the answer here:

The first session data unit (SDU) bufferful of return data is written to TCP socket buffer under SQL*Net message to client wait event.

If Oracle needs to return more result data for a call than fits into the first SDU buffer, then further writes for that call are done under SQL*Net more data to client event.

So, whether and how much of the SQL*Net more data to client vs. SQL*Net message to client waits you see depends on two things:

Amount of data returned to client per call
Oracle Net SDU size

A quick test with my Snapper performance tool illustrates this:

In one session run following:

SQL> set arraysize 1
SQL>
SQL> select rownum from dba_source;

    ROWNUM
----------
         1
         2
         3
         4
         5
         6
...many rows returned...

And I measure the waits in the other session:

SQL> @snapper out,gather=w 5 1 141

-- Session Snapper v1.06 by Tanel Poder

---------------------------------------------------------------------------------------------------------------------------------------------
HEAD,     SID, SNAPSHOT START   ,  SECONDS, TYPE, STATISTIC                               ,         DELTA,      D/SEC,     HDELTA,     HD/SEC
---------------------------------------------------------------------------------------------------------------------------------------------
DATA,     141, 20080210 14:59:55,        5, WAIT, SQL*Net message to client               ,          6750,       1350,     6.75ms,     1.35ms
DATA,     141, 20080210 14:59:55,        5, WAIT, SQL*Net message from client             ,       4668258,     933652,      4.67s,   933.65ms
--  End of snap 1

No SQL*Net more data to client waits show up above, as with arraysize = 1 setting Oracle fetches only 2 rows at a time. Therefore the returned resultset per fetch call always fits into the first SDU sized packet. Note that the reason why Oracle fetches 2 rows when arraysize is 1 is due to an OCI optimization which tries to be clever and proactively detect end-of-resultset conditions when fetching a single row - that way there’s a chance to automatically cancel a cursor and release its pins without an explicit cursor cancel or close call. This should have positive effect reducing shared pool fragmentation in some cases.

Anyway, now I’ll set the arraysize to 5000 and run the same SQL again:

SQL*Net message to client wait isn’t really what it’s thought to be

Thu, 07 Feb 2008 08:35:47 +0000

In a recent Oracle Forum thread a question came up how to use SQL*Net message to client wait events for measuring network latency between server and client. The answer is that you can’t use it for network latency measurements at all, due to how TCP stack works and how Oracle uses it. I’ll paste my answer here too, for people who don’t follow Oracle Forums: As I wrote in that reply, “SQL*Net message to client” does NOT measure network latency!

Oracle hidden costs revealed, part 1 – Does a batch job run faster when executed locally?

Mon, 04 Feb 2008 16:09:06 +0000

This series is about revealing some Oracle’s internal execution costs and inefficiencies. I will analyze few situations and special cases where you can experience a performance hit where you normally wouldn’t expect to.

The first topic is about a question I saw in a recent Oracle Forum thread.

The question goes like this: “Is there any benefit if I run long sql queries from the server (by using telnet,etc) or from the remote by sql client.”

In order to leave out the network transfer cost of resultset for simplicity, I will rephrase the question like that: “Do I get better performance when I execute my server-side batch jobs (which don’t return any data to client) locally from the database server versus a remote application server or workstation?”

The obvious answer would be “NO, it does not matter where from you execute your batch job, as Oracle is a client server database system. All execution is done locally regardless of the client’s location, thus the performance is the same”.

While this sounds plausible in theory, there is (at least) one practical issue which can affect Oracle server performance depending on the clients platform and client libaries version.

It is caused by regular in-band break checking in client server communication channel where out of band break signalling is not available. A test case is below:

Excellent article on Oracle 11g PL/SQL function result cache

Sun, 03 Feb 2008 07:39:25 +0000

I have so far avoided writing such pointer blog posts which only refer you to another article, but I have to do it with this one. Adrian Billington has written an excellent article on performance of Oracle 11g PL/SQL function result cache. His article is a good example of a thorough, well organized and well written technical content. I really enjoyed reading it and thanks to his thoroughness, he has just saved me some precious time doing that research on my own.

Can you write a working SQL statement without using any whitespace?

Mon, 14 Jan 2008 04:42:21 +0000

I read this post by Laurent Schneider yesterday. In the comment section Tom Kyte already explained what the issue was about, but I’ll expand this explanation a little.

The question was why should the apparently invalid statement below work? I mean there is no such column nor number as “1.x”), yet the statement works ok:

SQL> select 1.x from dual;

         X
----------
         1

The column header gives a good hint what happened above. Oracle has treated the X as the column alias.

Let’s remove the “x” and see:

SQL> select 1. from dual;

        1.
----------
         1

Now all works as expected, “1.” is treated as number ( 1. = 1.0 )

But the question remains, how come Oracle splits up the string “1.x” and decides that the “x” is the column and “1.” is the number part – considering that there’s no whitespace between the 1. and x?

The answer is that apparently the string tokenizer used by Oracle SQL parser is smart enough that it doesn’t rely only on whitespace for recognizing token delimiters. It is also able to use character class analysis for understanding where a literal ends and the next token (like column alias) starts.

So, it is possible to write valid SQL statements without using any whitespace at all. For example:

Expensive calculator…

Wed, 09 Jan 2008 09:53:10 +0000

Oracle has evolved over time to much more than just a plain relational database. One option is to use Oracle as an expensive calculator. When researching or demoing Oracle, it’s quite convenient to do number calculations directly on sqlplus prompt, especially if dealing with internals where lots of stuff is about addresses and offsets shown in hex. Here’s the script what I use for such purposes: https://github.com/tanelpoder/tpt-oracle/blob/master/calc.sql. It usually saves me couple of seconds every calculation as I don’t have to reopen the calc.

Why does Oracle parameter count change during session lifetime?

Mon, 07 Jan 2008 11:36:33 +0000

I was once asked a question, why does Oracle change its parameter count during session lifetime?

The question arose from the following observation that v$parameter shows more parameters after you adjust some hidden parameter value:

SQL>
SQL> select count(*) from v$parameter;

  COUNT(*)
----------
       288

SQL>
SQL> alter session set "_complex_view_merging"=false;

Session altered.

SQL> select count(*) from v$parameter;

  COUNT(*)
----------
       289

Looks like the parameter count was just increased by one!

“It sure seems like the hidden parameter don’t exist before they are actually modified”:

SQL> show parameter _unnest_subquery   (no rows returned)
SQL>
SQL>
SQL> alter session set "_unnest_subquery"=false;

Session altered.

SQL>
SQL> show parameter _unnest_subquery

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
_unnest_subquery                     boolean     FALSE
SQL>

So it seems like Oracle was “creating” the hidden parameter when it was modified.

This is not the reality though. All parameters for session are created during session startup and stored in shared pool.

The answer lies in the view text of GV$PARAMETER view. This example is from an 11g database, older versions like 9.2 do have less checks in the where clause. Note that the output is manually formatted for better readability:

Systematic application troubleshooting in Unix

Sat, 05 Jan 2008 11:55:59 +0000

How many times have you seen a following case, where a user or developer complains that their Oracle session is stuck or running very slowly and the person who starts investigating the issue does following:

Checks the database for locks
Checks free disk space
Checks alert log
Goes back to the client saying “we did a healthcheck and everything looks ok” and closes the case or asks the user/developer to contact application support team or tune their SQL

The point here is that what the heck do the database locks, alert log or disk space have to do with first round session troubleshooting, when Oracle provides just about everything you need in one simple view?

Yes, I am talking about sampling V$SESSION_WAIT here. Database locks, free space and potential errors in alert log may have something to do with your users problems, but not necessarily. As there are many more causes, like network issues etc which could affect your user (and the whole database), it doesn’t make sense to go through all those random “healthchecks” every time you receive a user phone call. Moreover, even if you identify that there is shortage of disk space or there are many database locks – so what? They may not have anything to do with the users problem.

The issue here is that still many people do not know about V$SESSION_WAIT which in most cases shows your problem immediately or at least points you to right direction (e.g. there’s no need to check for locks if your session is waiting on “log file switch (archiving needed)” wait – and vice versa). Even if “these people” have heard of V$SESSION_WAIT and may be able to drop this in during their job interview, they may not know how to use it in systematic troubleshooting context. Many hours of service downtime and user frustration would be saved if all DBAs knew this extremely simple concept of looking at V$SESSION_WAIT.

This blog entry is not about Oracle though, so I will leave this rant for a future blog post.

This post is about a similar problem in Unix world. Having been involved with resolving some serious production issues lately I have been surprised quite many times by the corporate Unix support people who seem to do behave in similar manner. For example, there is a user calling in saying that their scheduled Unix job, which normally takes 5 minutes, has been running for hours now. The “senior unix support analyst” will do following:

Perl version of Snapper

Fri, 28 Dec 2007 15:44:20 +0000

Wolfgang Breitling has written a snapper.pl script – Perl version of my original snapper.sql one. Check it out! As his script doesn’t rely on DBMS_OUTPUT for generating output, you can easily get real time continuous output with his script, without needing to wait until the script finishes or by using somewhat complicated DBMS_SYSTEM.KSDWRT tracefile output. Also, as his script doesn’t rely on DBMS_LOCK package for sleeping, so you don’t need rights on it.

Sqlplus is my second home, part 5: Reading the name of currently executing script

Tue, 25 Dec 2007 16:00:35 +0000

If you need to run and manage loads of sqlplus scripts which call other scripts, which call other scripts etc, then you are probably interested in the sqlplus APPINFO parameter shown below.

When you issue SET APPINFO ON in sqlplus, this makes sqlplus to automatically call DBMS_APPLICATION_INFO.SET_MODULE and set the MODULE value to sql script name which is currently being executed.

This allows you to easily pass the current script name info to Oracle, without the need to have a manual call to SET_MODULE in beginning and end of every script (along with some mechanism for storing the previous module).

A simple example is below. I used two scripts blah.sql and blah2.sql for my test:

C:\tmp>type c:\tmp\blah.sql

select sys_context('USERENV', 'MODULE') from dual;

@@blah2

select sys_context('USERENV', 'MODULE') from dual;

 C:\tmp>type c:\tmp\blah2.sql

select sys_context('USERENV', 'MODULE') from dual;


So, blah.sql reports the current module, then calls blah2.sql which reports current module and then returns back to blah.sql which returns the current module again.
SQL> set appinfo on
SQL>
SQL> @blah
SYS_CONTEXT(‘USERENV’,‘MODULE’)
01@ blah.sql
SYS_CONTEXT(‘USERENV’,‘MODULE’)
02@ blah2.sql
SYS_CONTEXT(‘USERENV’,‘MODULE’)
01@ blah.sql
SQL>

Looks cool!
From output above we can see the following things:



Oracle Security, Part 2: Your read only accounts aren’t that read only
Sun, 18 Nov 2007 16:44:56 +0000
Couple of years ago an interesting fact floated up in Oracle-L – a regular user with only SELECT privilege on a table can successfully execute a SELECT FOR UPDATE against it, locking all rows and even lock the whole table using LOCK TABLE command. Locking a table in exclusive mode would stall all changes and selects against that table – effectively hanging all applications using that table. Pete Finnigan wrote a review of the issue in this blog entry.


Sqlplus is my second home, part 4: Getting sqlplus parameter value into a variable
Tue, 06 Nov 2007 16:22:12 +0000
I’m having some busy times, so can’t blog anything more serious than another sqlplus trick (which likely has value only to some hardcore sqlplus geeks though).
Ever wanted to load a sqlplus parameter (like linesize, pagesize or arraysize) into a sqlplus define variable?
This can sometimes be helpful for customizing your everyday DBA scripts to output (or not output) some columns based on linesize. Or you may want to use the SQL error code somewhere in your script.


A simple interview question
Fri, 19 Oct 2007 16:20:28 +0000
Question: How to check instance parameter values in Oracle?
Answer: show parameter xyz
WRONG!
Answer: select value from v$parameter where name = ‘xyz’
WRONG!
These commands show the session level parameter values, which are separate from instance level parameters:
SQL> show parameter session_cached_cursors NAME TYPE VALUE ------------------------------------ ----------- -------------------------------- session_cached_cursors integer 20 SQL> select value from v$parameter where name = 'session_cached_cursors'; VALUE --------------------------------------------------------------------------------- 20 V$SYSTEM_PARAMETER is the view which shows instance level parameters (and these are what all new sessions inherit)


How to resolve SQL object and column names all the way to base tables and columns in Oracle?
Sun, 16 Sep 2007 05:23:08 +0000
Update: As this original article is from 2007, there are better options available in the modern times – for example DBMS_UTILITY.EXPAND_SQL_TEXT as explained by Maria Colgan’s blog entry here.
If you have been involved in tuning SQL code which you have never seen before, you are probably familiar with the challenges of understanding what the code is trying to do. This can be especially time consuming when the SQL references lots of views, which reference views, which reference more views etc. So there may be a large information gap between the SQL statement (like select * from some_crazy_10_level_view) and the actual execution plan (referencing 10s of tables, with evidence of query transformations).
So unless you see something really obvious from the execution plan, you need to start mapping the SQL query and view texts back to the physical base tables which Oracle eventually has to access. This can be a tedious and boring (!) process.
The good news is that in Oracle 10.2+ there’s a hidden parameter that can do this mapping task for us.
Update: It looks like 9.2.0.8 patchset also has this parameter (so I guess later patchsets of 10.1.0.x have it as well now)
Let’s see an example:
I create a view on a view to illustrate the point:
SQL> create view myview as select * from all_users;

View created.


Now let’s set that parameter _dump_qbc_tree to 1 and run a query against the view:
SQL> alter session set "_dump_qbc_tree"=1;

Session altered.

SQL> select count(*) from myview;

  COUNT(*)
----------
        31


Now let’s look into the server process tracefile:
*** ACTION NAME:() 2007-09-16 12:19:57.500
*** MODULE NAME:(SQL*Plus) 2007-09-16 12:19:57.500
*** SERVICE NAME:(SYS$USERS) 2007-09-16 12:19:57.500
*** SESSION ID:(146.1984) 2007-09-16 12:19:57.500
QCSDMP: -------------------------------------------------------
QCSDMP:  SELECT: (qbc=2B8D1C28)
QCSDMP:    . (COUNT(*)) (opntyp=2 opndty=0)
QCSDMP:  FROM:
QCSDMP:    .MYVIEW
QCSDMP:      VQB:
QCSDMP:        SELECT: (qbc=2B8D163C)
QCSDMP:          .USERNAME
QCSDMP:        FROM:
QCSDMP:          .ALL_USERS
QCSDMP:            VQB:
QCSDMP:              SELECT: (qbc=2B8CAF78)
QCSDMP:                U.NAME (USERNAME)
QCSDMP:              FROM:
QCSDMP:                SYS.TS$ (TTS)
QCSDMP:                SYS.TS$ (DTS)
QCSDMP:                SYS.USER$ (U)


Here it is, the query text generated directly from parse tree, showing the base tables regardless that they had been hidden behind multiple views.
Also there’s few interesting things to note:


Advanced Oracle Troubleshooting Guide, Part 3: More adventures in process stack
Wed, 05 Sep 2007 16:50:32 +0000
…or rather thread stack as nowadays decent operating systems execute threads (or tasks as they’re called in Linux kernel).
Anyway, stack trace gives you the ultimate truth on what your program is doing, exactly right now. There are couple of but’s like stack corruptions and missing symbol information which may make the traces less useful for us, but for detailed hang & performance troubleshooting the stack traces are a goldmine.
So, I present another case study – how to diagnose a complete database hang when you can’t even log on to the database.


Sqlplus is my second home, part 3: Colored selections in Windows XP command prompt
Sat, 01 Sep 2007 05:55:23 +0000
Whenever delivering some Oracle training or running a demo at a conference, I’ve always liked to use the Windows command prompt version of sqlplus.
One reason of course is its easy command line history navigation capability ( press F7 in cmd.exe after entering few commands to see why ).
Another reason is that whenever I want to highlight some part of sqlplus output, I can just drag a selection rectangle around that text. In other words I can “mark” the text – drawing the attention there. Of course as the selection rectangle is really meant for copy & paste operations only, it has several limitations. It’s not persistent, whenever I continue typing, the text “marking” will disappear.
Windows XP has introduced a really cool feature to cmd.exe, which anyone doing presentations involving some command line tool output will appreciate!
Basically XP allows you to persistently select and color command prompt output!
An example of what I’m talking about is here:


Sqlplus is my second home, part 2: Running SQL scripts from remote locations using HTTP
Thu, 30 Aug 2007 16:11:20 +0000
As you probably already know, the Session Snapper has been designed to be a very easy-to-use performance tool. It is especially useful in database environments where there are no decent performance tools pre-installed and available.
Snapper doesn’t require any setup, all you need is to log on to the database using sqlplus and download snapper.sql script to your computer.
Well, actually the second part is not required, as Oracle sqlplus allows you to run scripts from http and ftp locations!
C:>sqlplus "sys/oracle@ora92 as sysdba"

SQL*Plus: Release 10.2.0.3.0 - Production on Thu Aug 30 23:00:10 2007

Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.

Connected to:
Oracle9i Enterprise Edition Release 9.2.0.8.0 - Production
With the Partitioning, OLAP and Oracle Data Mining options
JServer Release 9.2.0.8.0 - Production

SQL> @https://github.com/tanelpoder/tpt-oracle/blob/master/i.sql

Tanel's sqlplus http test...
https://github.com/tanelpoder/tpt-oracle/blob/master/i.sql

USER                           SYSDATE
------------------------------ ---------
SYS                            30-AUG-07

NAME
---------
ORA92

INSTANCE_NAME    HOST_NAME
---------------- ----------------------------------------------------------------
ora92            WINDOWS01

You should be *very* sure that noone can change the scripts on the server without your knowing!!!

SQL>


Cool stuff or what? :)
Let’s see how this relates to everyday DBA life…


Operating systems are lazy allocating memory
Tue, 28 Aug 2007 15:02:24 +0000
There was a discussion about whether Oracle really allocates all memory for SGA immediately on instance startup or not. And further, whether Oracle allocates memory beyond the SGA_TARET if SGA_MAX_SIZE is larger than it.
It’s worth reading this thread first: http://forums.oracle.com/forums/thread.jspa?threadID=535400&tstart=0
I will paste an edited version of my reply to here as well:


Advanced Oracle Troubleshooting Guide, Part 2: No magic is needed, systematic approach will do
Sun, 26 Aug 2007 17:40:48 +0000
There are two ways for diagnosing problems:
Checking for the usual suspects and hoping to find a matching one Following a systematic approach Checking for the usual suspects and hoping to find a matching one The first approach relies on previous experience (both in particular subject area/technology and about the context/environment the problem occurs). For example if a patient comes to doctor complaining about pain in chest, then for doctor (and also for the patient) it would definitely be beneficial to know more relevant info about the patient - the context.


Oracle Session Snapper, part 2: Getting most out of Snapper
Sun, 26 Aug 2007 04:02:36 +0000
The main design goal of Session Snapper was that it should not require any changes to be made into database.
And to achieve this goal, I was even willing to sacrifice some functionality.
So, for example there is no sorting capability in Snapper output. It would have been easy to create an SQL Type to database, use that as session statistics storage and query results out using an order by on statistics delta column – giving you (probably) most significant resource consumers first. But I didn’t do this as it would have violated the no-database-change-whatsoever design goal. (This problem could however be solved using manual sorting in PL/SQL code as done in Adrian Billington’s variation of runstats utility: http://www.oracle-developer.net/utilities.php )
The second design goal was that snapper should work with as little privileges as possible. So far it requires execute access only on DBMS_LOCK, DBMS_OUTPUT and read access on few V$ views.
However if you run the snapper using “out” option then it will use DBMS_OUTPUT.PUT_LINE for sending data back to the client. Unfortunately the client (SQLPlus in this case) calls DBMS_OUTPUT.GET_LINES to retrieve the output only when the previous database call has finished. Of course this makes sense, because otherwise some asynchronous call capability to Oracle server would be needed, which either has to interrupt the server process somehow for processing the GET_LINES, or another server process should be started for output feed.
Anyway, we don’t have such functionality in Oracle, so this means that if you run snapper in a loop, you will get the output only when the loop finishes and returns control to sqlplus. This is not good enough for having continuous real-time view of your session(s) performance.
Those who have looked into the header or sourcecode of my Session Snapper script have seen that if you use “trace” option and tail -f on that tracefile, you can get continuous real-time Snapper output to your screen (in addition to having that info saved in logfile). This functionality uses DBMS_SYSTEM.KSDWRT() procedure though. This means again that in order to get real-time output, you need to either grant execute priv on DBMS_SYSTEM to your monitoring user or run Snapper as SYS. Both those options violate the design goals of Snapper and may be unacceptable in real life production systems.
So, to combat that, I wrote a script which still gives us real-time Snapper output without need to access DBMS_SYSTEM.


Oracle 11g internals part 1: Automatic Memory Management
Mon, 20 Aug 2007 18:15:51 +0000
This is my attempt for getting cheap popularity out of recent Oracle 11g release. This is not going to be another Oracle 11g new features list, I’ll be just posting any of my research findings here, in a semi-organized way.
The first post is is about Automatic Memory Management. AMM manages all SGA + PGA memory together, allowing it to shift memory from SGA to PGAs and vice versa. You only need to set a MEMORY_TARGET (and if you like, MEMORY_MAX_TARGET parameter).
You can read rest of the general details from documentation, I will talk about how this feature has been implemented on OSD / OS level (or at least how it looks to be implemented).
When I heard about MEMORY_TARGET , then the first question that came into my mind was that how can Oracle shift shared SGA memory to private PGA memory on Unix? This would mean somehow deallocating space from existing SGA shared memory segment and releasing it for PGA use. To my knowledge the traditional SysV SHM interface is not that flexible that it could downsize and release memory from a single shared memory segment. So I started checking out how Oracle had implemented this.


Oracle Session Snapper – real-time session-level performance stats for DBAs
Sun, 19 Aug 2007 17:05:41 +0000
A post by Jonathan Lewis inspired me to finally complete my version of the Oracle session performance snapper script, which main characteristics are

it reports Oracle session level performance counter and wait information in real time
it does NOT require any database objects to be created

If you are a DBA or consultant working on ad-hoc performance issues, you will like it!
Are you familiar with following situation?

(Monday morning)
Customer: Hey, we need your help! We have serious performance issues in our production environment. It’s a database with x000 online users, several parallel data feeds, continuous batch jobs and reporting going on.
Users have started experiencing occasional bad performance and some batch jobs as well.
You: Ok, lets see it. Do you have any performance monitoring tools installed?
Customer: Yes, we have Xxxxx Xxxxxxxxx installed which shows us a nice green or red light on big screen depending whether there are any problems or not.
You: What color is it showing now?
Customer: Green
You: But you still have the performance problem you described earlier?
Customer: Yes
You: Do you have ANY performance monitoring tools installed?
Customer: Well, we also have a statspack snapshot taken every morning and evening to capture the business workload.
You: Um… Ok… you know forget about it. Can I create a small package to capture some useful performance info on the problematic sessions?
Customer: Yes, but we need to put the DDL scripts through the change review board which gathers every Thursday… but we can’t wait that long!
You: Can we enable tracing?
Customer: Enabling tracing is a change and all changes must go through review board. Also, we don’t really know which exact sessions are affected, it just happens for seemingly random ones… and we can’t just trace every session, can we?
You: Ok, give me a sqlplus window and Excel, we’ll figure something out.
…And now follows a tedious manual SQL execution and copy & paste exercise from various V$ views for getting some meaningful performance information out of Oracle.

Well, not anymore, because The Oracle Session Snapper is in town!
If you know vmstat for Unix, you know it reports you various system level statistic counter deltas over a period you choose.
Well, the Oracle Session Snapper output looks somewhat similar, but it reports you session level deltas of Oracle v$sesstat counters, wait events and starting from 10g also session time model statistics.
All info can be reported in real time, without a need for running and timing multiple SQL scripts and manual calculation of deltas.
And the key unique point of the Session Snapper is – it does not require creation of any database objects, thus no changes in the database at all! Everything is done from within a sqlplus script or anonymous PL/SQL block.
This means that you will be able to get quick session-level performance snapshots even in heavily change-controlled environments, where no object creation whatsoever is allowed without going through a long process.
You can read all the usage details from the script header, but here’s one example of its output:
Tanel@Sol01> @snapper out 1 3 475

-- Session Snapper v1.03 by Tanel Poder (  )

--------------------------------------------------------------------------------------------------------------------------------------------
--        SID, SNAPSHOT START   , SECONDS  , TYPE, STATISTIC                               ,         DELTA,      D/SEC,     HDELTA,   HD/SEC
--------------------------------------------------------------------------------------------------------------------------------------------
DATA,     475, 20070820 01:17:47,         1, STAT, session logical reads                   ,         88232,      88232,     88.23k    88.23k
DATA,     475, 20070820 01:17:47,         1, STAT, consistent gets                         ,         88233,      88233,     88.23k    88.23k
DATA,     475, 20070820 01:17:47,         1, STAT, consistent gets from cache              ,         88232,      88232,     88.23k    88.23k
DATA,     475, 20070820 01:17:47,         1, STAT, calls to get snapshot scn: kcmgss       ,           556,        556,        556       556
DATA,     475, 20070820 01:17:47,         1, STAT, no work - consistent read gets          ,         87677,      87677,     87.68k    87.68k
DATA,     475, 20070820 01:17:47,         1, STAT, table scans (short tables)              ,           139,        139,        139       139
DATA,     475, 20070820 01:17:47,         1, STAT, table scan rows gotten                  ,       7429598,    7429598,      7.43M     7.43M
DATA,     475, 20070820 01:17:47,         1, STAT, table scan blocks gotten                ,         87676,      87676,     87.68k    87.68k
DATA,     475, 20070820 01:17:47,         1, STAT, buffer is pinned count                  ,           138,        138,        138       138
--  End of snap 1
DATA,     475, 20070820 01:17:48,         1, STAT, session logical reads                   ,         87779,      87779,     87.78k    87.78k
DATA,     475, 20070820 01:17:48,         1, STAT, consistent gets                         ,         87772,      87772,     87.77k    87.77k
DATA,     475, 20070820 01:17:48,         1, STAT, consistent gets from cache              ,         87772,      87772,     87.77k    87.77k
DATA,     475, 20070820 01:17:48,         1, STAT, calls to get snapshot scn: kcmgss       ,           552,        552,        552       552
DATA,     475, 20070820 01:17:48,         1, STAT, no work - consistent read gets          ,         87210,      87210,     87.21k    87.21k
DATA,     475, 20070820 01:17:48,         1, STAT, table scans (short tables)              ,           138,        138,        138       138
DATA,     475, 20070820 01:17:48,         1, STAT, table scan rows gotten                  ,       7389897,    7389897,      7.39M     7.39M
DATA,     475, 20070820 01:17:48,         1, STAT, table scan blocks gotten                ,         87211,      87211,     87.21k    87.21k
DATA,     475, 20070820 01:17:48,         1, STAT, buffer is pinned count                  ,           136,        136,        136       136
--  End of snap 2
DATA,     475, 20070820 01:17:49,         1, STAT, session logical reads                   ,         87580,      87580,     87.58k    87.58k
DATA,     475, 20070820 01:17:49,         1, STAT, consistent gets                         ,         87587,      87587,     87.59k    87.59k
DATA,     475, 20070820 01:17:49,         1, STAT, consistent gets from cache              ,         87587,      87587,     87.59k    87.59k
DATA,     475, 20070820 01:17:49,         1, STAT, calls to get snapshot scn: kcmgss       ,           552,        552,        552       552
DATA,     475, 20070820 01:17:49,         1, STAT, no work - consistent read gets          ,         87046,      87046,     87.05k    87.05k
DATA,     475, 20070820 01:17:49,         1, STAT, table scans (short tables)              ,           138,        138,        138       138
DATA,     475, 20070820 01:17:49,         1, STAT, table scan rows gotten                  ,       7375781,    7375781,      7.38M     7.38M
DATA,     475, 20070820 01:17:49,         1, STAT, table scan blocks gotten                ,         87041,      87041,     87.04k    87.04k
DATA,     475, 20070820 01:17:49,         1, STAT, buffer is pinned count                  ,           137,        137,        137       137
--  End of snap 3

PL/SQL procedure successfully completed.


The output contains 3 x 1 second snapshot of session 475 doing heavy nested looping. Note that even though the CPU time used was not updated, the logical IO counts for that session had still increased.
So this tool can be very valuable diagnosing what’s going on when the session seems to be 100% on CPU doing something.


Sqlplus is my second home, Part 1: HTMLizing your sqlplus output
Tue, 07 Aug 2007 14:14:23 +0000
I have not managed to post anything for a while, but I intend to make it up by starting this series of posts made specially for Oracle enthusiasts, geeks and maniacs among us. Here I plan to post the coolest Oracle stuff I’ve just found out and some of it may actually be useful to you!
Lets start. This post is about removing the last major problem with sqlplus in everyday database and application administration work.
I would say the commnd line sqlplus, combined with its script execution and Windows cmd.exe’s command history navigation capabilities, is a very powerful and fast tool for database administration and troubleshooting. This is of course if you use a set of database administration scripts, either downloaded from some reliable source or accumulated over the years of working with Oracle ( you do have such scripts, right? ;-)
Now to the main weakness of sqlplus: I may have scripts carefully formatted for my screen size, however when adding more columns, I run out of screen width. When working on an application data quality troubleshooting task, I need to run some quick ad-hoc queries. Or run a query which just returns lots of data.
What usually happens in such cases is illustrated very well with the output of following query:


Session-level statspack
Sun, 24 Jun 2007 07:23:26 +0000
Statspack is a useful tool for easily gathering and reporting some Oracle’s historical workload statistics. However it has its limitations and problems:
One of them that in past it used to record only Oracle statistics, measured from inside Oracle. This made Oracle and people using Oracle ignore other crucial statistics like OS workload below Oracle. For example statspack may show you heavy latch contention as the performance problem, while looking at vmstat output one would see that the server just has been heavily overloaded with numerous other jobs (like multiple backup, export and compress jobs overrunning their run-window) and the latch contention is just a symptom of CPU starvation.
This has been somewhat addressed by adding V$OSSTAT to Oracle 10g and statspack now gathers this info as well. Now we need people to start looking into those stats.
Another problem with statspack is which is described in Dan Fink’s blog, about cursor-level CPU accounting limitations before 10gR2.
However neither of those problems are the main limitations of statspack. The main problem is that statspack samples it’s statistic snapshots from system level views like V$SYSSTAT and V$SYSTEM_EVENT. If every single session in your database is performing exactly the same (kind) of operation, then system level aggregates might be quite OK to get an overview what’s going on in the database. But from the moment you start having different kind of sessions (e.g. OLTP vs batch vs data feed vs report) in your database, then system level aggregates used by statspack do heavily distort what the reality looks like for specific sessions.
There’s another gotcha – by default statspack excludes the “SQL*Net message from client” wait event from main report section, stating that it is not important to look into idle events. However by that we will throw away an important piece of information when diagnosing end-to-end performance, the end user experience. The bad thing is that even if we did include the SQL*Net wait event in our reports (by deleting the corresponding row from STATS$IDLE_EVENT), then we would still have no idea how many of those gazillions of system-wide SQL*Net client wait seconds do belong to our specific session or user.
Statspack doesn’t measure session-experience, it just gives you a system-wide aggregate, which cannot be translated back to individual session statistics (just as you can not convert a hash value back to original value – most of the information is just lost!).
How to get session-level detailed overview of database performance, with historical reporting capability then?
Oracle 10g addresses this partially with ASH and AWR and DBMS_MONITOR’s selective statistic sampling (Search for V$SERV_MOD_ACT_STATS).
While ASH and AWR are both awesome products, they are not usable in Standard Edition nor before 10g ( you may want to check out Kyle Hailey’s ASH simulator instead ) and for using them you need to pay a separate license fee.
So let me introduce my attempt to solve those problems ( NB! Free stuff!!! ;):

The poor-man’s version of Automatic Workload Repository: It’s called… um… Semi-Automatic Workload Repository :)
The clever-man’s addition to statspack, which takes session-level snapshots: In other words, Sesspack.

The idea is very simple, just take snapshots of V$SESSION_EVENT and some V$SESSTAT (and V$SESS_TIME_MODEL in 10g) statistics and store those in a repository just like statspack does. Snapshots are taken using sesspack.snap_xyz procedures. As a parameter to these procedures I can pass a specific SID, a group of SIDs, a Oracle db username a OS-username or whatever filtering fields can be found from V$SESSION.
Once you have taken your snapshots when running the workload, you can run a report which calculates statistic and wait event deltas between snapshots and that it.
A demo?


A gotcha with parallel index builds, parallel degree and query plans
Sat, 23 Jun 2007 08:54:10 +0000
Reading the following article about PARALLEL hint by Jonathan Lewis made me remember a somewhat related gotcha with parallelism.
Often when creating (or rebuilding) an index on a large table, doing it with PARALLEL x option makes it go faster – usually in case when your IO subsystem is not the bottleneck and you have enough spare CPU capacity to throw in.
A small example below:
Tanel@Sol01> create table t1 as select * from all_objects; Table created.


My version of SQL string to table tokenizer
Tue, 19 Jun 2007 18:48:10 +0000
This one’s a short post on a fairly random topic as unfortunately I don’t have time today to come up with anything deeper :)
I needed to come up with a delimited string to table tokenizer for an Oracle development project. There are quite a few examples out there how to do that, including Adrian Billington’s www.oracle-developer.net and the http://technology.amis.nl/blog/?p=1631.
So far the simplest solution I had seen was using a bunch of INSTR’s, SUBSTR’s and DECODE’s in a CONNECT BY loop.


Advanced Oracle Troubleshooting Guide: When the wait interface is not enough [part 1]
Mon, 18 Jun 2007 13:38:16 +0000
Welcome to read my first real post on this blog!
If I ever manage to post any more entries, the type and style of content will be pretty much as this one: Oracle problem diagnosis and troubleshooting techniques with some OS and hardware touch in it. And internals! ;-)
Nevertheless, I am also a fan of systematic approaches and methods, so I plan to propose some less known OS and Oracle techniques for reducing guesswork in advanced Oracle troubleshooting even further.