This article is not meant, in any way, shape, or form, to put down Windows Admins. Every group has bad eggs, and this is just me writing my experience with the "bad eggs." Please keep in mind, Part 2 of this article will go through my experiences with good Windows Admins, as well as bad. Not all Windows admins are morons, just like not all Linux/Unix admins are the l337.
Introduction
I have been a professional Linux/Unix Admin for about 7 years now. I actually started off as a Junior Admin at a Linux company. The experience there, taught me a lot, but also got me spoiled. You were made fun of if you used Windows.
At that company, every desktop in the building was running some distro of Linux. Except for one Exchange Server the Development Team used when writing an email client for Linux, and Solaris that would communicate 100% to exchange (yes before evolution did it, and before exchange 2000).
So basically I never had to argue about Linux/Unix stability, ease of use, etc.
I left the Jr. Admin Job to be a full-time Admin at a web development company. The company was awesome, everyone was super nice and to this day I say it was the best job; however it did have its rough areas.
Most of our projects were PHP, Java or ASP, so we ran apps like Tomcat, Websphere and Oracle Application Server (basically a souped up apache with Java built in).
I got hired to take care of all Unix/Linux machines, but little did I know that I was the only Linux/Unix Admin among 5 Windows admins...yes count em 5!
My First Incident with a Windows admin
Our windows admin always gave me a hard time about Linux. He always told me it was less stable, and much harder to work with.
To prove my point, I challenged him to an up-time contest. So he set up his own DNS Server to be a slave to the Master DNS I had just set up using Linux.
After about 120 days I get a call from our Windows admin bragging that my Linux DNS server went down. I was in shock. How the heck did this server go down? I had to have a look.
I went to the KVM switch to bring up my server. When the screen came up, there was a Windows log in prompt.
I was in disbelief. I actually checked all the connections to make sure they were correct.
Come to find out, my clearly labeled LNXDNS-server01, was the victim of a Win2k Server install by the same Admin I had challenged in the up-time contest.
He felt really bad when I asked if that was his way of trying to beat me in the uptime contest.
If your curious, the contest was a blow out. The Windows machine got a virus and failed around day 140. My Linux server went more than 400 days before I had to disconnect it when the company moved to a new location.
Actually, the first incident wasn't to bad. It was actually funny, more than anything.
More Issues
[before I begin this section, I'd like to stress that my jobs has always been great because of the people I work with]
One of the on-going issues was working with the 7 people in charge of the technical team. All of them were cool in there own way, but only one understood Unix/Linux. It was actually a little worse with him because he knew the basics because he, had been a Unix administrator many years ago.
Lets go into some of those issues:
Support
Now, this exact item can be a pain in the ass if your not running a commercial Unix or Linux. I am a firm believer in FreeBSD and some of the free Linux's. However, they do not have commercial support, this caused a lot of unneeded panic. This is how I handled the situation:
Boss: Do our CVS, FTP, PHP/Apache servers have support?
Me: No, they are running a very stable OS called FreeBSD, and they are backed up on a daily basis using a custom perl script I wrote that sends the backups to our fileserver via FTP.
Boss: Thats good, but why don't we have support contracts on these?
Me: Uh, not to sound harsh; but my job is to support these machines correct?
Boss: Yes, but still
Me: Well heres the thing, FreeBSD doesn't have official commercial support. You can hire someone to do this, but your basically going to get someone like me (with more or less knowledge) and pay a lot more money.
Boss: O.k, I see now. I am sorry about the misunderstanding, I am so used to having support on our Windows machines that it seemed odd.
Now this boss was very cool, and very open minded. The great thing is this made him think. Now we had 5 Windows admins, and we payed for every OS ($1000+) and also paid for the top level support contracts with Microsoft.
Now think about that. What the hell is the point of a Windows Admin with all that support? You would think that we could hire a person with basic Windows knowledge to take care of these machines and call support when needed. Or we could have the guys we already have, and not pay for the support. The Windows world is a very confusing place. I know my boss was thinking hard about these things after that conversation.
Virus Scanners
I have had this conversation with every boss except the ex-Unix Admin boss. The conversation usually went like this:
Boss: Windows Admin-A tells me we don't have a virus scanner running on any of our Linux or Unix machines. I need you to find how much this will cost us, and give us a estimate on the time it will take.
Me: There is a reason why we don't run a Virus scanner; because its not needed.
Boss: Why is that? (Keep in mind the Windows Admin did this knowing we didn't need it, think he thought it was funny)
Me: Well none of the Linux machines are fileservers or Email Servers
Boss: So? They are still exposed to the outside world, we have Windows machines that aren't Email or Fileservers and they need and have virus scanners.
Me: Thats true, but Apache, Oracle, Websphere etc are not easily exploited as IIS or any other Windows service. Virus's, just don't exist in the Unix/Linux world like they do in Windows. When a Linux/Unix machine is exposed to the world, I do my job by locking it down and turning off easily exploited services like telnet, talkd, ftp , dns etc. The only thing we have to worry about are rootkits, and I have plenty of measures to stop them and to alert me if we catch one.
Boss: How much did that cost us, and what is the software called.
Me: All of the software was free. I use chkrootkit and Tripwire.
Boss Uh, O.k. Sounds good (has a confused look on his face)
Thes situations were not to bad, but it becomes annoying. It really makes you want to beat your head against the wall when your boss who happens to run the technical side of the company ask questions like this. Man I need to get into management somehow.
Getting Excluded because your a threat
The incident I'm about to write about could of been avoided if I had been involved at the start. Anyway this is a combination of issues with other admins, and power.
It all started when we got a new client, and I mean a big client. The client was to buy all the hardware, software and support as suggested by us. Now, the project is 100% Linux, so you would think they would bring in the Linux Admin to help make the decision right? Well, they didn't. The current "manager" I had, was put in charge of the group. Sort of the head administrator.
Anyway the sales guys brought him in, (they didn't know the difference between OS's and admins) and asked him to help. Usually when this happens we will all have a meeting a discuss it, not this time.
The first thing that went wrong was the sales people brought in Experts, who brought in there own experts. The project was a Oracle/Oracle application design. We needed a production environment, patch, dev, test, standby and failover! IBM came in and brought their friends VMWare along. In the end, my manager came out with 3 machines, 3 copies of VMWare ESX, and 5 copies of Redhat AS 2.0.
Had I been consulted, the company could have saved about $15,000 (yes thats right) and ended up with the correct hardware/software combinations of 3 machines, 1 copy of VMWare, and 3 copies of Redhat AS 2.0.
So I can let that slide as not my problem. However it was a problem when the manager laid out what we are going to do:
Machine one (PROD): Install VMWare ESX and install 1 virtual OS. (yeah, I know whats the point of running vmware with only one OS)
Machine two: Install VMWare ESX and install 4 OS's (patch, dev, test and failover) - Not to bad, but don't like failover setup this way
Machine three: Install VMWare ESX and install 1 OS (again, the point of VMWare is?)
I questioned this, and he basically gave me a hard time. Saying it was the best way and that I had no clue what I was talking about because I have never used VMWare ESX. O.K whatever, I went around him. His Boss pretty much ignored me too.
Now the kicker, Production is running one virtual OS, and needs the Oracle database, and Oracle application running on it at once. According to my boss, the best way to do this is to cripple the OS by putting it on the second layer (layer one is VMWare, layer two is OS). Then, lets install 6Gigs of ram, but only allow the OS to access 3.5Gigs.
Now, you newbies out there may not know this but X86 hardware has limits, and Vmware has them to, so each process is limited to 3.5Gigs of ram (VMWare treats each OS as a process), thus we get limited use of the ram.
As you can guess, when we went live we saw a huge performance problem. We ended up installing another OS and Split the DB and Application. This helped out a lot, but could of been avoided from the start if we just didn't run VMware on it.
We could of had one machine running both apps and db, taking full advantage of 6Gigs of ram and be running at full speed un-virtualized. But, we were stuck with the setup. We couldn't destroy VMWare, and just go full Redhat because uptime for this client was important. Luckily the split was easy to do with VMWare.
The fact that we bought 4 licenses of Redhat for 1 machine is pathetic. Think about this, when you buy Redhat your paying for a license for one machine. So 1 license should work. Oh well, again not my problem.
In the end, the client turned out to be OK and the boss, manager and I ended up leaving. My boss, never saw me as a threat he was just doing the wrong job. He should of been managing another team. My manager, however, was very threatened by me. He would get new projects and never let me in on any of it. He would exclude me from meetings and all kinds of stuff.
So watch your back. If your company runs both Windows and Linux/Unix, and you know your way around both, be prepared for people to dislike you.
Oh no, I now have to take care of Windows too!
Just like every company back in those days, we had layoffs.
Everyone knew that I was a Unix/Linux Admin, but people later found out that I was capable of PHP coding and even Windows administration.
As we lost our PHP developers, and our ASP deployment Admins, I got stuck with the ASP Deployment. Nothing to complicated handling this stuff. However, 1 year later they laid off all admins and decided that since I could administrate both Windows and Linux that I will do both.
Being in charge of all machines has it perks and downsides. A big perk was that I had full control, and with this we (meaning me) decided it would be best to replace all Windows servers that could be replaced with Linux or FreeBSD.
I started small by moving our FTP servers over to BSD. Later, I converted all of our SourceSafe Servers over to CVS. The developers loved me after converting to CVS! Our firewall was nothing complicated, so I moved this over to. Man everything was Linux/Unix except our email. And there was no way I would get around this.
With all the perks mentioned above I must mention the downside... That Exchange 5.5 server was hell.
I knew nothing about it. Adding new accounts at first was a total pain in the butt. On top of that the storage area was running out of space, and people refused to delete email (small company, so they got away with it).
So, I did some searching one day and found that there was a exchange Log directory with over 15 gigs of used space! OMG, wtf. These logs were old, I mean like 3+ years old.
So, like any Unix Admin I did some cleanup. I deleted half of the logs and thought everything was cool. 20 minutes later email is down, WTF. Apparently the logs, were not just logs but journaled data or something odd like that. Now that is weird.
Microsoft needs to learn to name things differently. Logs should be logs, and should be there just for history. 25 hours later (yes I worked 25 hours straight), after talking to MS morons for 7+ hours (keep in mind they charge for this) I got email back up with almost nothing lost. This was a huge nightmare.
First off, we had no backup server. We were running so low on money that we couldn't afford to fix the backups. Second, this email was important, we are talking 5 years of email for every employee. After this, I made sure to read up on any changes. I won't even go into the nightmare of me converting from Exchange 5.5 to 2000. Granted the machine after that is more stable, but it was hell converting.
People who get that you know your stuff
This can be good and bad. As a Unix/Linux Admin, people know that I am capable and will take advantage, especially the Windows admins.
Now, we had a situation where a file needed to be transfered via FTP from a Windows box to a Linux box. Well apparently the Windows Admin thought it would be better if a script was written on the Linux machine. So I had to write a script to go and get files, rename them etc. I also had to do the opposite, no big deal but as you can see the Windows Admin sort of proved me right by basically saying its easier to do this stuff on Unix/Linux machine than it is on Windows
. In fact you can see the simple script I wrote in our forums.I had this kind of stuff happen all the time, and I gladly did them. Why? Because it shows my co-workers that Unix/Linux are better, and that I am capable. So its a win/win situation. It may suck to do the work, but you know what that's my job even if the other guy is lazy. I have been at my company for 5+ years now and lived through close to 5 rounds of layoffs; wonder why I am still here while all the other admins got laid off?
The a$$hole Winblows admins
I've only worked with one really bad Windows Admin, but most of them have done stupid shit that just pissed me off.
As a Linux/Unix Admin you will be told frequently how the system you work with is old and outdated, complicated, and just plain stupid when compared to the great thing they call Windows. I have learned that you just have to nod, and not argue with them because its pointless. For every proof you show them, they will come back with some BS MS FUD. You could be mean, and say the system is easy, it's just that your so stupid you need to click shit to make a server run...but that wouldn't be good
So do like me, and just not fight it. I have tried to fight it in the past and it was just pointless and got no where. Don't fight win fan-boys of any kind. And for the record, Linux fan-boys are just as bad as Windows fan-boys.The Dumb-asses
OK, so a good Linux/Unix Admin will be jack of all trades when it comes to the computer world. Most will have some basic knowledge of just about everything. I want to explain a situation I ran into a few years back that made my jaw drop and got me so freaking frustrated.
We lost power, and I came in along with our Windows Admin at the time to check in on a 2 servers that didn't come up automatically. One Windows and one FreeBSD.
Windows Admin goes to his and to his surprise the Windows machine is going in reboot loop. BIOS, OS start, reboot, BIOS, OS start reboot etc, basically a file system error occurred that was fixed by booting into safe mode and running scan disk I believe.
So we go to my machine and its hung at the BIOS level. I turn it off and turn it back on, same thing. Right after the memory count, and before the SCSI initialization it hard locks. Hrmm, that's weird. I show Windows Admin who laughs at me. This is how the conversation went:
Win Admin: Damn unstable Linux, if that thing was running Windows you would of been able to easily fix it.
Me: Huh (keep in mind, this guy has A+ certification so he should know how a computer works)
Win Admin: System won't even start to boot, Linux really bit the dust that time ha ha.
Me: Hey smart guy, you are aware that it hasn't even tried to touch the Master Boot Sector?
Win Admin: Like I know that Linux crap, you don't have to worry about that crap in Windows.
Me: OMG (I start laughing).
Win Admin: Whats so funny?
Me: Let me get this straight, your A+ certified correct?
I then go on to explain that the issue is not a OS, but at the hardware level. And I also taught him what the master boot record was.
Now all Windows admins are not this way, but in my experience most of them don't know half about OS's or Hardware then your average Unix/Linux user. So keep this in mind. We are a rare breed, in a dumbed down world. Don't blame the Win Admins, blame MS for making crappy and un-open software so they can make money off of businesses (anyone and their dog could get MS certified).
No comments:
Post a Comment