Kuwait Data Bank

Introduction

 I've been pitching the idea of Kuwait National Datacenter to government officials and parliament members since 2017, in a holistic manner, covering migration of applications, learning center, audit requirements, security, ...etc. -- unfortunately, those efforts fell on deaf ears.

Fast forward to 2021, and I got the chance to work with a fantastic group of volunteers for Kuwait Foundation for  the Advancement of Science (KFAS) to create something more specific: Kuwait Data Bank -- an entity that would hold data from all of Kuwait's government entities and government-owned companies, to do Data Analytics and Data Science.

The group of volunteers comprised of multiple disciplines; experts in law, business management, organizational structure, information technology and security. A friend of mine & I were covering the information technology (IT) and security aspects.

KFAS gave us 1.5 months, that we extended to 2.5 months max to get the initial draft out, and we were done in Oct or Nov 2021. We're now in discussion with KFAS to see how to proceed & hopefully we get to see this project go live at some point!

Project Scope and Goals

We've checked regional and international open data projects, and almost all had very limited sets of samples of data, over inconsistent timespans, and sometimes one time shot kind of data only. Our project's aims are ambitious and exceed anything we've checked.

1. Initially, we'll focus on 1-5 critical reports to the Council of Ministers for decision making support.
2. Gradually, as we sanitize data, and find a source with the most truthful data (or combined sources), then we aim to make data pulling and reporting mainstream and real-time.
3. Data and reports will be available/accessible in this order:
1. Council of Ministers
2. Expand access slowly to government entities in need of help in accurate decision making
3. Universities in Kuwait
4. Public access inside of Kuwait
5. International access to data and/or reports or reporting services
4. Leverage latest technologies of graphics card acceleration and Massive Parallel Processing (MPP) databases in software (non-appliance) to keep things agile and portable.

Data Access & Analysis Methodology

1. Start slow with as few sources of data as possible to deliver the critical reports
2. Deploy data masking & replication connectors to the various databases at the sources
3. Anonymize data at the source, then replicate to our organization's repository/repositories
4. Sanitize data and compare accuracy with help from people at each data source, initally
5. Run Machine Learning models on highly parallelized data access databases
6. Produce reports or dashboards with results of multiple ML models and compare results
7. Initially, those reports will be private and delivered only to the Council of Ministers or KFAS, but gradually, the platform will expand to allow real-time access to reports, and then later, our anonymized data sources
8. Data access & reporting may be monetized to help the platform grow and become self-sustaining, in addition to providing services for companies to run analytics on their data, or using our data sets.
9. Legal aspects of data access, anonymization & privacy, and cooperation from government entities have been addressed in our report/proposal, but I'll not get into that here.

Privacy & Anonymity

1. A primary design aspect is to respect privacy and anonymize data at the source, before it's sent to our repositories/databases
2. Example: if we're to take everyone's full address, we'd remove the house number, but keep the area, and area's block number
3. If our systems get compromised, there will be no personally identifiable information (PII) that would cause personal risks
4. We believe that leaving the data masking (anonymization) in the hands of each government entity giving us access is probably the best approach, so that we will never be able to make changes to what data we receive, without manual intervention from the data sources (government entities)

There's a lot more to the project, but I'll stop here and then maybe revise things once we see how the project will move later.

It's an ambitious project, which is why we need to grow gradually and cater for specific needs that help the country's decision makers in making critical decisions and answering crucial questions, before making a decision.

Oracle Licensing and Virtualization Restrictions

Disclaimer

The information here represents my personal findings using published documents from Oracle. It doesn't represent my legal opinion. I am not a lawyer. Take this information and fight for your right as a consumer/customer and demand an official response from Oracle by email, not verbal.

Introduction

I've had numerous encounters with customers citing Oracle sales people stating that virtualizing Oracle DB on VMware is not supported, and that the licensing of the entire physical host's cores, or even the entire cluster's cores is mandatory, and in a nut shell: this is NOT entirely true and can be circumvented.

The information below is based on Oracle's legal documents and licensing documents and guidelines. Check the references for the links and details.

References are denoted with numbers. When you see #1 it means see reference number 1 at the end of this post.

Executive Summary (TL;DR)

The Oracle partitioning guide is not a contractual document and Oracle strictly states it's for educational purposes only. Therefore it cannot use it to impose how customers should partition their environments or systems.

Excerpt from Oracle's "Oracle Partitioning Policy" document

Details and Resources

The only contractually obligating documents from Oracle are:

• Technical Support Policy document
• Processor Core Factor table
• Oracle License and Service Agreement (OLSA)  / Oracle Master Agreement(OMA)

Terminology and Concepts

Alright, let's get into details, and one step at a time to provide a full picture. First things first:

License Types

Oracle DB is licensed in different ways, depending on its edition:

• By number of users using the connected application or whose data are saved in the DB
• By number of CPU sockets. A socket is a full physical processor, regardless of the number of cores inside it.
• By number of CPU cores. This is the case for the Enterprise edition of the DB. Cores refer to the number of physical cores in every CPU socket installed in the server. Threads are considered logical cores, and you do not license those; only the physical cores.

Audit Compliance

Before moving forward, let's discuss audit compliance. Oracle audit team can request to audit your environment. You have the choice of not allowing them, but they might come back later with government officials to enforce it, or if you open a support ticket, they'd do an inspection anyway to see whether you're eligible for support or not.

If/When you do allow Oracle's audit team to run an audit, it's essential to agree on a scope and limited time to do the activity: i.e., clusters 1, 2 and 3 and the activity is to not exceed 2 weeks maximum. This is to be stated in the legal document you'll be signing prior to them starting the audit activity.

Oracle will ask you extract virtual machine (VM) activity logs, where they've been and where they've moved for X number for months. It's also important to limit the period: 1-3 months should be sufficient for any audit activity.

If you do not impose such limits, Oracle can keep asking for extended periods to run their scripts, and try to find at least 1 case of deviation to impose penalties on you.

Some sales people will scare you that you're violating Oracle terms and will be subject to penalties of millions of USD.
As long as you're complying with the rules below, and have done the settings properly and have full log of all VM activity (in vCenter) to prove that the VMs haven't moved beyond the licensed hosts and cores, no one can penalize you.

If they continue to harass you, ask them to send you an official email, and once you receive it, send it to Oracle's USA legal team. You'll receive a note from them acknowledging that you're in full compliance.

Feel free to reach out to me and I'll help you reach the right people within Oracle. For planning and designing help with your setup to make sure you're compliant prior to deploying the Oracle workloads, I can offer this within Kuwait only. If you're outside of Kuwait or the Arabian Gulf region, I suggest you contact a vendor selling x86, another selling IBM POWER and let both give you 5-year Total Cost of Ownership (TCO) studies including Oracle software license costing.

License Core Factor

Oracle applies different ratios of licenses needed for each core depending on the processor/CPU being used in the servers [#1]. This is called Core Ratio, and usually for Intel mid-range processors (Intel E5-2400, E5-2600, Xeon Silver and Xeon Gold), the core factor is 0.5.

For Oracle's own SPARC CPU, the core facor for M5, M6, M7 and M8 CPUs is 0.5. This is in bid to push for its own hardware and provide a full solution.

For higher-end processors (Intel E7-4800, E7-8800, Xeon Platinum, and IBM POWER), the core factor is 1.0.

Processor Choices and License Calculation

If your application vendor says they need 10 cores, you have to ask them to specify which processor and model have they benchmarked their database workload on.
It's unfortunate that many software vendors benchmark their workload once (say 2012 for example), and then keep using the same hardware requirements on newer systems, which means customers (you) end up with an extremely over-sized solution.

Why? Because 10 Intel Xeon E5 cores in 2012 are equal to about 6 Intel Xeon Gold cores now (rough estimate). The same applies to any processor brand, such as IBM POWER and Oracle/Sun/Fujitsu SPARC. The enhancements vary, but the idea is the same: do not believe the software vendor's requirements unless they tell you which hardware was used to do the benchmark.

If the application vendor says you need 10 cores on Intel Xeon Gold, then you need to purchase 10 (cores) x 0.5 (Xeon Gold core factor) = 5 Oracle Enterprise DB core licenses.

If the application vendor says you need 10 cores on IBM POWER9, then you need to purchase 10 (cores) x 1.0 (IBM P9 core factor) = 10 Oracle Enterprise DB core licenses.

The above does NOT mean that POWER core factor is more expensive than Intel, because the performance of 10 cores on Intel is less than 10 cores on POWER. That's why Oracle assigned POWER processors a higher core factor. However, I will NOT discuss which processor to choose in this post, to not derail from the topic of licensing.

The advice I give all my clients is: Choose the platform that gives you the best Return on Investment (most cost effective) and reliability. Make sure to always factor in cost of software and hardware for 5 years for your solutions, including maintenance, support and subscription costs.

Virtualization/Partitioning Types

Virtualization allows you to simultaneously run multiple virtual machines (VMs), each with its own operating system (OS), on the same physical server.

Oracle treats hypervisors (virtualization engines/software) differently, mainly as a sales tactic (politics) and not for technical differentiating factors (though some technical factors exist, but the main drive remains to push sales into their direction).

Oracle's list of supported virtualization and partitioning technologies, VMware's vSphere is not listed, for political reasons, but it's fully technically functional and support is provided as follows:

• If the problem is already known, Oracle will provide support.
• If the problem is unknown, Oracle require you to reproduce the issue on a physical server.
• Some of my customers said they've had tickets open on supported platforms for months, while Oracle support engineers threw the blame on Microsoft Windows, and Windows support engineers threw the blame on Oracle. So you can imagine how this might turn on an unsupported platform.
• VMware openly states that it will provide full support for Oracle software running on VMware's platform, so you contact VMware and they'll use their in-house Oracle certified support engineers. See the links in references for details on what VMware covers. [#4]
• Oracle sales people might tell you you'll never get support, but that's a lie. Ask them to email you their claim, and then you can escalate that. 99.99% they won't dare email you since it's illegal to make such claims.

Licensing types based on virtualization: Oracle licensing states that you need to license every CPU core that's used by the database. That's easy to do on Unix platforms, but requires additional configurations on x86 (AMD/Intel) platforms.

x86 (AMD/Intel) systems

As x86 systems are considered commodity servers, they don't offer a function to isolate specific cores for specific workloads/VMs. However, with VMware vSphere or Microsoft Hyper-V hypervisors, you can assign specific processor cores to always be used by a specific VM. Hyper-V calls it CPU Pinning. vSphere calls it CPU Affinity.

Keep in mind, you need to also restrict which hosts are able to run these virtual machines, in addition to the CPU core affinity. On VMware vSphere, when enabling High Availability, a VM will restart on a different host if the original host lost power, therefore you need to set cluster policies to have the VMs run on specific hosts only, even in cases of host failures.

Remember the audit section above? This is why you need to setup such restrictions here.

Example: You have a VMware cluster of 4 hosts, each host has 2x 14-core Intel processors (28 total). You have/need Oracle Enterprise Edition DB effective licenses for 8 cores = 16 Intel cores licensed (0.5 core factor license for Intel mid-range CPUs).
You want to run 2 instances of Oracle DB as virtual machines, each with 8 cores (4 effective core licenses).

You can easily create a Host Affinity rule in VMware's Distributed Resource Scheduler to restrict the DB VMs to specific 2 hosts in the cluster, and edit the VM settings to specify 8 cores in each host as part of the CPU Affinity settings. This way you lock the 2 VMs to 2 specific hosts in the cluster, and each VM to specific CPU cores.

There is no need to buy dedicated servers for Oracle with the CPU cores matching the license. You do need to license any host cores that will run Oracle instances (2 hosts to have high availability -- if one VM goes offline, the other is still functional. Do not power on the other one.

If you need to do maintenance on one physical server: power off one of the 2 VMs, and carry on your maintenance, then power it on when the host is ready.

Unix (POWER/SPARC) systems

Such systems allow cores to be pooled/grouped and specific workloads can be restricted to certain cores. This is known as Hard Partitioning.

I am not very familiar with SPARC systems, so my example(s) will be for POWER: if you have a machine with 2 CPUs, 10 cores each, for a total of 20 cores, you can create a Shared Processor Pool of 6 cores and restrict all Oracle DB VMs/LPARs to run on that pool only. This allows you to license only 6 cores, and the VMs will share those 6 cores.

It's often that customers buy dedicated core licenses for each DB they create, however, in many times when we did utilization analysis of those VMs and DBs, the CPU utilization was much lower than the assigned values, however the customer had a huge number of total cores licensed for Oracle DB!

A better approach is to create a pool for the DBs, and let the VMs use the cores from that pool. Additionally, on POWER, it's possible to allow a VM to have 2 cores, but increase number of cores if needed, then scale back. In such a scenario, it will never exceed the restriction impose by the pool, so you always remain within the license boundries.

The above setup helps with one part of the audit, but when it comes to moving VMs/LPARs around different physical hosts, the same rules apply: you have to license the physical cores where the VMs run. So, if you have 2 physical hosts for High Availability, there are few ways to do the setup:

License Options

Licensing One VM only

Oracle licenses per installed instance. If you create a VM with an OS and install Oracle DB on it, you have to license it, even if it's offline/powered off.

To license one VM only in HA setup, you have to use storage replication, or connect both hosts to the same storage, such that at any time, only one VM instance exists on the servers. When you need to failover to your 2nd site or host, you do the job manually and import/power on the VM on the 2nd host, as long as it's powered off and removed from the 1st host.

Additionally, you have to disable Live Partition Mobility, vMotion or any function that allows VMs to move between hosts in the same cluster.

This is true for both x86 and Unix.

Licensing Multiple VMs

If you wish to use Oracle DataGuard or Real Active Cluster (RAC) to guarantee data consistency using application/DB-level replication, then you need to license at least 2 instances and setup the replication scheme on the DB level.

Additionally, you have to disable Live Partition Mobility, vMotion or any function that allows VMs to move between hosts in the same cluster. You license cores on physical hosts that have the VMs running. If you do want the VMs to move to other hosts, you'll need to license all hosts permissible for movement.

This is true for both x86 and Unix.

As you can see, these legal restrictions are not technical limitations, but only to enforce customers to pay more licenses and/or enforce Oracle's own ecosystem onto the customer to further leverage more purchases in the future.

Pitfalls

As you can see above, when using x86 systems, there's some added overhead on the operations team to make sure the VMs always remain in compliance when doing daily operations and maintenance jobs. It's easier to do things when having Unix systems and maintain compliance, but then you need operations people with Unix skills.

A mistake of 1 person in operations could put you out of compliance. If you run a small company, you may be better off with buying dedicated physical servers for Oracle workloads (but end up with many physical boxes). If you're an enterprise with many Oracle workloads, I suggest moving away from them if possible, and if not, go with a Unix environment that gives you flexibility and is able to reduce your overall cost on software licenses.

References

1. Oracle Processor Core Factor Table
2. Oracle Partitioning Policy
3. Supported Virtualization and Partitioning Technologies for Oracle DB and RAC
4. VMware Support for Oracle on vSphere
5. Understanding Oracle Certification, Support and Licensing on VMware Products
6. Oracle Misinformation on VMware

Tech Talk: Knowing NoSQL

Web Master, developer of the underground, and master of mischief @burhan will be talking on NoSQL databases, on Wednesday Feb 22nd 7:30 PM - 9:30 PM at Kuwait IT Society in Roudha area.

Event details: on Failbook (Arabic); on Google+ (English).

The event is organized by our favorite local code exporter, @ahmad_alibrahim.

TPSA: When Everything Goes Right

I was going to write about a networking story, but what happened yesterday registered as a day to never be forgotten, hence this write-up.

TPSA: The Perils of System Administration -- A series of true stories about system administration. The first story is here.

Yesterday, Thursday April 16^th, we had a scheduled maintenance job on our production servers. The cluster wasn't functioning as it should and yesterday's tasks were aimed at rectifying the problems once & for all, and update the software installed.

This was our time-plan for each task on the list:

1400 to 1410	Shutdown SAP on DB and APP Server
1410 to 1440	Take a database backup Shutdown database
1440 to 1500	Take a Filesystem backup (All Filesystems on both nodes) Changing mointpoints for High Availability Cluster switchover testing
1500 to 1700	Shutdown SAP and database Perform Kernel Upgrade on SAP CI Start the database and SAP on the CI Instance ONLY (Not the APPS) Perform technical testing on CI Check the mountpoints exe Start SAP on APPS Perform technical testing on APPS
1700 to 1730	Import the ST-A/PI Patch 01L_ECC600 Change SAP Parameters based on the document Review the parameters on CI and APPS (Memory and Work Processes) Restart the SAP CI and APPS Perform technical testing on CI and APPS

All tasks were easy and planned out with all members of this properly set:
- ERP software consultant (joined by his colleague later)
- ERP offshore support consultant
- AIX Unix consultant from IBM Kuwait
- Myself

We grabbed lunch around 1230 and some snacks and headed to the Head Office (H.O.).

@1401: ERP Applications were stopped

@1406: We started a full offline backup of the production database

@1437: The backup reached its final stage, then threw an error, stopped, and gladly deleted the backup

*NO!! THIS IS NOT THE TIME!! PLEASE!*

After being stunned and depressed for 2 minutes, I thought of stopping the database and starting again, using the user db2ehp I did that and when I tried to start the database again, it threw an error!

*JAWS DROPPED -- LAAAAAAAAA!!*

We faced the same error a few days back due to some user profile changes, so we had to run the command "/usr/opt/db2_08_01/instance/db2iupdt db2ehp" -- I ran that command and it puked an error ...

*NOOOO!! WHY WHY!! IT WAS FINE WHY NOW!! I HATE YOU!!*

I navigate around and go to the home directory of the user db2ehp to check the environment variables and profiles, only to find out that these files have been corrupted and turned to binary garbage

*SHOCKED*
*EYE TWITCHES*

I ftp to the secondary database node, looked for similar files, then decided to copy the whole directory and renamed the files to match the host name of the primary node.

It didn't work for some reason, even though all the environment variable scripts had proper values...

I called the company responsible for the support on our backup software (TSM) and in about half an hour their consultant provided me with a command line to recover a certain directory to a specific location.

Then we restored from our backup software that whole directory: "/usr/tivoli/tsm/client/ba/bin/dsmc restore /home/db2ehp/ -subdir=yes /bkfs2/restoreyaman/"

@1620: All files were recovered, and now were able to switch to the user db2ehp properly.
I run the comman "/usr/opt/db2_08_01/instance/db2iupdt db2ehp" and then start the database and it worked!

*YESSS!!*

Database backup started again and we waited till it finished.

DB backup finished and we wanted to take a backup of the filesystems through TSM. Going back & forth, we eventually invoked it manually through TSM, but it timed out & didn't work. Error in communication through TCP/IP, it said!

*THE IP IS WORKING! I CAN PING AND LOGIN WHY CANT YOU!!???? BLOODY $##^%^@*

@1743: The IBM engineer arrived and suggested that since the directory to be changed is very small, then just copy the directory somewhere else. We copied the directory we were going to change with "cp -pR " to preserve permissions, to a remote filesystem.

Now that everything is ready to be changed, the offshore support was contacted and their consultant logged in to our server through VPN and did the required changes.
The changes were simple: One of the filesystems was part of the cluster resources and its mount point was incorrect. We simply had to change the mount point from "/db2/db2EHP to /db2/EHP/db2ehp"

After that step was done, the IBM dude synchronized the changes between the cluster nodes on the IBM AIX Unix cluster. That was half the work -- now we just needed to make sure that we can failover back & forth, then we will proceed with patching the ERP software to the latest version.

@1805: We crashed the primary node, to simulate a failover from the primary DB node to the secondary, tested our ERP software, and it was working.
Upon crashing the primary node, it was booted up again from the management console and left to come up. Meanwhile, we were checking that the DB was working properly on the 2nd node and that the ERP software was able to communicate with the DB and everything was fine.

@1815: The primary DB node didn't come up yet. Fishy. Upon checking on it, it seemed to be stuck, so we restarted it again.

@1825: The machine didn't come up after 10 minutes, which was very suspicious... after checking the management console, it was stuck at code: "0557" -- Some Googling away and to our luck, the message meant problems with the filesystem....

The system couldn't boot because from what it seems, the root filesystem (/) is corrupted, hence the operating system can't load.

*EYE TWITCHES*
*WHY ???? WHYYYYY??? WHY NOW? WHY ME? *

I'm quite disappointed, since I have never expected to face such an issue with AIX and on a p5 series machine. Even the IBM dude was shocked.

The IBM engineer said he can proceed with the procedure we found here:
http://www.docstoc.com/docs/2801670/AIX-BOOTING-PROBLEM -- page 5

But he said that since his job isn't support, it would be better for us to call IBM's support line and log the call, and whatever instructions they give, he'll execute them.

I call IBM's branch in Kuwait and dial the extension which usually takes me to their support in UAE. No one answered... I called 2 more times, without avail.

The IBM dude called a colleague and he gave him another extension for off-hours support. We called that extension and someone picked up! (OH JOY)

I told the support dude my company's name & that we're from Kuwait and he asked me what was the problem & to log an issue by sending an email. I sent the email.

@1953: I received an email from him asking for my company's name, again.

I reply to it and wait for another 10 minutes. Then I call the extension again and ask him what is going on, he said that our support contract had expired in 2005.

*HUH?!*

I tell him that we bought the hardware in November 2007!! And that the support contract for that didn't expire yet!! He insisted that there was no data on their end to support my claim and we argued for a good 10 minutes.
The IBM dude with us interfered and said that my claims are correct and that he was present during the purchase and commissioning of these boxes, but the dude at UAE said if their database doesn't show such a thing, there's nothing he can do.

@2024: He emails me with instructions to contact IBM Europe, and that if they are to help us, they will charge $360 an hour for a minimum of two hours!

According to all the links we found on Google, the error code points at a corrupt filesystem, so we know what's the problem exactly, and there was no point in contacting or paying for IBM Europe.

@2035: we proceed with loading the first DVD of AIX and boot from it to recovery mode on he primary node, and follow instructions in the document above.

All filesystems we corrupted. ALL of them.

*I looked at the IBM dude and said: I'm this close to sit in a corner and cry*

The AIX guru started fixing them one by one, and all got repaired (including root filesystem), except one: /home, which contains the startup and environment scripts for the database...

*CRAP! but at least the root filesystem is sane!*

We reboot the machine and enter the recovery mode again and run fsck (ilesystem check) again to make sure all the filesystems are fine now .. all were, except /home -- it's a goner. Can't be recovered anymore. The LVM partition was corrupt beyond recognition.

The IBM engineer proceeds to make sure that the root filesystem is bootable, on both disks, by issuing the respective commands to write the Master Boot Records to them, and other information.

We exit the recovery mode, for the machine to boot in normal mode and we look anxiously at the error code display... as soon as it passed and the system started to come up, we jumped in joy and hugged.

@2050: Now that we have most of the filesystems working, we decide to back them all up on a DVD (sysback).

@2105: The backup to the DVD failed. Apparently the unix box only likes DVD-RAM media. Luckily, there was an option to take a backup over the LAN to TSM.

@2115: We then proceeded to destroy the corrupt filesystem and its evil logical volume, then create a fresh one and import the /home directory contents to it from TSM: "/usr/tivoli/tsm/client/ba/bin/dsmc restore /home/ -subdir=yes /home/"

@2135: After restoration was done, we took another full backup of the root volume group (which includes the new /home filesystem).

Then we rebooted the primary node, to make sure that the filesystems persisted. It didn't come up after 5 minutes... When checking, the IBM dude had forgotten to abort booting from CD, at which the screen was stuck at. Exiting from that menu, the machine booted normally.

@2145: We rebooted one more time, just to be sure, and everything went fine. At this point, we no longer needed the offshore support nor the IBM engineer since their job was done. The offshore support logged off & the IBM unix guru left, with many warm thanks from me.

Then we proceeded to patch the ERP software; basically, it's just a compressed file with the new binaries and an installer script. We already had the stuff uncompressed on a remote filesystem (NFS), so we just mounted that, renamed the old directory (exe) to "exe_old", then copied the files to the proper location.

It should be noted that the filesystem which we applied the patch to, is exported as a network filesystem (NFS) to other nodes.

As the ERP software came up on the primary node, we started it on the 2nd node, but it crashed...

The ERP dudes tinkered around and found out that all nodes except the primary ERP one are using the old files (pre-patch) !!!

*EYE TWITCHES*
*WHAT THE!!*

We found out a few minutes later that for some reason, the NFS mount is still pointing at the old directory, which we renamed! So, it seemed like even if you rename the directory, NFS would keep track of that!! (Maybe we should've stopped NFS before doing the renaming?)

I proceed to re-export the NFS directories, in an attempt to refresh any links to the directories. It didn't work, and now other nodes are getting an error for this particular NFS mount:
"df: /sapmnt/EHP/exe: A file, file system or message queue is no longer available."

*EYE TWITCHES*
*NO MORE PROBLEMS, PLEASE!!! LET US FINISH AND GO HOME!!*

I stopped the NFS service & started it again. No use. Deleted the old exported directory settings from NFS, and add it again, then restart NFS. No use.

At the end, we decided to try moving out the files from the old directory to a temporary one, putting the new files in the renamed dirctory "exe_old" then renaming it back to "exe"

It worked!!!!! And I laughed hysterically, not believing what happened and the "solution"

During all this, we had to take down the cluster resources before modifying anything, since all nodes point at the shared NFS filesystem, even though it's not part of the cluster resources!!!

We had to shut & start the cluster services about 5 times, till we figured out the solution above.

We brought up all the systems and the ERP guys applied application-level patches & plugins (yes, more of them...)

@0057: I took the ERP dudes to their hotel and I went home.

The Perils of System Administration

Whenever you join a company, visit the IT department and you'll find a group of guys calling themselves Systems Administrators, or Systems Engineers (depending on their rank).
These are the guys that keep your IT services running, and even if you see them slacking, playing chess, hide-and-seek, making fun of users, or sleeping on the desk, you can almost always count on them when one of the servers go down; they will stay at work and use toothpicks to keep their eyes open, until your precious(ssss) services are up & running.

I have come across two kinds of admins: Those who have ethics and those who don't. In time of crisis, you can tell which is which, in case the unethical was a cunning fox during the casual days of duty.

Ethics dictate that you state clearly what you know & what you don't, take responsibility for your actions, be loyal to your employer, don't abuse your power, and do your job as you should.

I'll share a couple of stories here to further show the dedication, demand and abuse that IT administrators are subjected to.

Sleepless Nights: Data? What Data?

Three weeks ago we had a scheduled maintenance task starting at Thursday 1400 hours (2 PM) till 1800 (6 PM). The scope of this task was to fix our Database Server cluster where our Enterprise Resource Planning (ERP) software works. This means HR, Finance, Warehouses, and Sales are all dependent on it.

We stopped the system at 1405, took an offline full backup of the database before working, then we proceeded to verify the backup to make sure it's consistent.

At 1435, the backup was done, and we proceed with the offshore support of the ERP system to fix the cluster problems on the secondary/standby node.

One thing lead to another, and we ended up staying till 2200 (10 PM) and planned to continue working on Friday starting at 0800, hoping to finish before lunch time.
The offshore guys were still logged in through VPN from India and continue to dig around for a few more hours.

On Friday I was at the Head Office (HO) at 0750, contacted the offshore support and we picked up from where we left. Around 1100, we got both nodes to work, and we did 2 failover tests and the database worked fine, until we switched back to the primary node.

Everything just went down the hill from there...

The database entered an infinite loop and entered recovery mode. What is recovery mode, you say?
Well, it crashes, then comes up again trying to start, then crashes, and so on.
These continuous cycles caused the error dump filesystem to fill up, which caused another crash at a higher level, stopping the recovery cycle and ending with a non-working database server.

Around 1600, we were still trying to bring the database up after investigating many error logs of the database and the operating system.

A few more futile attempts were made to run the database, after increasing the size of the error dump filesystem.

Around 2100, we realized that our database has been corrupted. No more data. No more business.

Enter panic mode.

We knew we had a safe full backup after the business closed, so we won't be losing any changes. Now, it was all about recovering the database, make sure the ERP software is working, then we could sleep.

I forgot to mention that since we couldn't leave the place, I had a friend of mine bring us lunch to work, and that was the only meal we had that day. THANKS HISHAM!!!

Anyway, we raised a support ticket to the ERP software vendor (SAP) with the highest priority possible and they called me within 30 minutes from Germany. They verified that it is indeed a top priority problem and they assigned one of their elite support guys to help us.

Around 0200, Saturday, we decided to dump the existing now corrupted database, and import the backup.

At 0300 Saturday, we had wrapped up the unfinished work of the cluster failover task, and were just happy that the data and the systems are up & running for the business to use, since my company works on Saturdays.



More stories to come...