Tracy's blog

I don't like Dell

Tracy R Reed — Wed, 16 Jan 2013 19:05:00 -0800

Why I hate Dell servers:

Every Dell machine which my clients have purchased and paid big money for has
caused problems. I'm not very happy with my Dell experience overall. Note that
I didn't choose Dell. I recommended against it. The organization footing the
bill chose Dell. I get to install and manage the Dells and get paid for my time.
But I would prefer to get paid for time being productive and not fighting
the hardware.

On the bright side: Future employers: I have LOADS of experience with Dell
hardware and have found workarounds for many of their warts! :)

Now, pardon my rant as I blow off some frustration:

First was the sales process. I don't want to have to haggle for a week to get a
good price. But that's what we did. And the price came down a fair bit.
Probably not as much as the time it cost us though. I don't want to have to pay
extortionate prices for RAM or hard drives either. I hate that a 6 bay hot swap
machine comes with blanks instead of drive trays. If you want more trays you
have to buy them from Dell with marked up Dell drives. I understand wanting to
only support drives known to work but tell me what model number that is so I
can get them wherever I want and give me drive trays with the machine. If the
machine has 6 drive bays it better come with 6 drive trays in those bays. It's
games like this...

We bought a memory upgrade from Dell for our 2970's to bring them up to 32G of
RAM. After installing the RAM and rebooting the computer said the memory
configuration was not optimal and prompted me to press F1 to continue. It would
then boot up just fine. But I can't have the servers requiring human
intervention for a reboot. So I had to figure out what the problem was. I
called Dell support and it turned out that the BIOS did not properly support
32G without a BIOS upgrade.

We were told they supported up to 32G when we bought them but it turns out the
BIOS they were shipped with didn't properly support 32G. So...that's broken at
time of purchase in my book.

Every one of our Dell servers has required a BIOS upgrade. The 610's would
spontaneously reboot after a couple of months in operation at first. They all
did it. Then I upgraded the BIOS. Now it has been at least 9 months since that
happened and I hope it is cured. Now standard practice is a BIOS upgrade right
out of the box. I really don't expect to ever have to upgrade BIOS in a server.
If I do that means it was broken when I bought it. Bugs don't appear by
themselves over time, they are there at time of shipment. Not only that but
there is mainboard BIOS firmware, DRAC/BMC firmware, and RAID controller
firmware all in need of updating. That's just too much stuff requiring
post-sale fixing.

As for the process of doing the BIOS upgrade there is room for improvement.
First, I am happy that there are Linux executables for doing this. It used to
be that only DOS binaries were distributed for stuff like this. But the process
for obtaining and executing the upgrade is rather obtuse.

The first step is to download the BIOS update. I was given this url by tech
support:

http://support.dell.com/support/downloads/download.aspx?c=us&cs=555&l=en&s=biz&r
T&osl=en&deviceid=11598&devlib=0&typecnt=0&vercnt=11&catid=-1&impid=-1&formatcnt
362396

Wow. That's a mess of a url. I don't like to have to download the BIN file on a
desktop or laptop and then scp the file over to the Linux server as it is
inconvenient. We don't run a web browser or any GUI desktop at all on our
servers as it is a waste of resources and not best practice. But I pretty much
need one to copy and paste that url and navigate the webpage it points to.

It would be nice if Dell provided a simple direct download link. Or at least
didn't wrap the Download button with a javascript function. If I am on my
laptop I like to right click the download link on my laptop and select "Copy
link location", then paste the url into an ssh terminal on the server and pull
the binary directly down to it. Currently when I right click the download
button and copy the link I get:

javascript:downloadslink('http://ftp.us.dell.com/bios/PE2970_BIOS_LX_4.1.1_1.BIN
verDownloadManager.application?c=us&l=en&fileid=362790&fileloc=ftp://ftp.us.dell
alse','PE2970_BIOS_LX_4.1.1_1.BIN');

Ugly and unusable. However, from this I can see that the actual path to the
file is:

http://ftp.us.dell.com/bios/PE2970_BIOS_LX_4.1.1_1.BIN

So on the server I can do:
# wget http://ftp.us.dell.com/bios/PE2970_BIOS_LX_4.1.1_1.BIN
and download the file directly onto the server.

Much more convenient. I can even type that by hand without copy and paste if I
really have to.

The firmware upgrade executables never work on CentOS. This is a gratuitous
limitation since it is functionally the same as RHEL. I can usually just change
one line in the shar file and make it work but I shouldn't have to.

When I execute this BIN file it produces an error indicating that it wants
another program called lockfile to be installed on the system. It took me a
while to remember this program. I had seen it before somewhere. Turns out it is
part of the procmail mail filtering program which we do not normally install
onto our servers. Most people shouldn't be installing that unless they need it
as part of a mail server. I had to install it to get the file to run.

Then I find that I also have to install compat-libstdc++-33-3.2.3-47.3.i386.rpm
but at least the BIN file gives me a useful error directing me to install it.
This is only needed for executables compiled against the old C++ library.
Moving to the newer one (why wouldn't they just use straight C for a firmware
installer?) would remove a barrier to getting the firmware update done.

This is pretty sweet:

Continue? Y/N:y
Executing update...
WARNING: DO NOT STOP THIS PROCESS OR INSTALL OTHER DELL PRODUCTS WHILE
UPDATE IS IN PROGRESS.
THESE ACTIONS MAY CAUSE YOUR SYSTEM TO BECOME UNSTABLE!
.../tmp/PE2970_BIOS_LX_4.1.1_1.BIN-6001-9159/./UpdRollBack: error
while loading shared libraries: libxml2.so.2: cannot open shared
object file: No such file or directory
.
The update failed to complete

Oops...looks like it is complaining that it can't find libxml2.so.2 so I gess
there is some XML nuttiness in this firmware somewhere. Installing libxml2 with
yum resolved that.

Then the firmware update installed and I rebooted. Yeay.

So that covers firmware.

The RAID card management tools leave MUCH to be desired as well. As far as I
can tell, the MegaCli package is the way to manage the PERC from the command
line in Linux. To work with it you have to hunt down the
MegaCli-1.01.39-0.i386.rpm tools since the tools are proprietary to LSI and
don't ship with RHEL.

[omstorage stuff is the right way to do this but that isn't clear at first]

Then you RPM install it and go looking for the software it installed. MegaCli
is rarely used. Only when setting up disks. They didn't call it megacli or
something I might remember. They called it MegaCli64 (case sensitive) which is
installed in /opt/MegaRAID/MegaCli/MegaCli64.

Then you have to figure out how to use it.

# /opt/MegaRAID/MegaCli/MegaCli64
Fatal error - Command Tool invoked with wrong parameters

hmm...ok

# /opt/MegaRAID/MegaCli/MegaCli64 --help
Invalid input at or near token -

hmmm

# /opt/MegaRAID/MegaCli/MegaCli64 -h

whoah! This gets you a massive amount of cryptic command line options with no
explanation as to their purpose. I have pasted the output here:

http://pastebin.ca/1968565

This is their idea of "help". I'm a command line commando of 20+ years and this
scares even me! It would have been nice if they at least tried to make it work
somewhat like the Linux mdadm command or at least provided some examples of
common use cases etc. Because of the oddity of this command various people out
on the net have compiled "cheat sheets" to help poor souls like me figure out
how to use this thing:

http://tools.rapidsoft.de/perc/perc-cheat-sheet.pdf

Usually I avoid using this command and just reboot the server into the BIOS and
configure the RAID card from there but often it is not a convenient time for a
server reboot. I also avoid it because it is so complicated and one wrong
command can lose all of the data in the server. Yes there are backups which I
would really rather not have to restore.

I needed to add a couple of disks on the fly and did not want to reboot. The
command line I seemed to need and response it gave me was:

# /opt/MegaRAID/MegaCli/MegaCli64 -CfgLdAdd -r0 [32:4] -a0
Adapter 0: Configured the adapter!!

Not a very reassuring response. Configured it how with what? It would be nice
if it said "Added virtual disk number 4 as a RAID 0" since that is what that
command told it to do.

Using the command:

/opt/MegaRAID/MegaCli/MegaCli -LDInfo -Lall -aALL

I was able to verify that it had in fact created virtual disk number 4 as a
RAID 0. However, I didn't have a file to work with in /dev representing the
disk. The operating system simply refused to see the disk so that I could
actually do something with it. I spent some time trying to figure out why but
couldn't come up with a solution. So I called tech support.

Dell tech support people are always friendly and, thankfully, seem to be US
based. That is a big help when the tech support person and I are yelling
instructions at each other over a noisy datacenter on a mobile phone. They
don't always have the solution, though. In this case with the RAID controller
I had added a disk and was trying to make it usable/visible to the OS. The guy
first guessed that I needed to partition the disks. I explained that the disks
were not visible to the OS to be partitioned. Then he guessed at some MegaCli
commands which were not useful. Eventually I had to get off the phone and head
out for an appointment. Later I got an email explaining that he had the
solution: I needed to run partprobe. That command finds partitions. You can't
find partitions on disks which you can't see. Way off the mark. Eventually it
became more convenient to reboot the server. So that is what I did and the
disks appeared. Problem solved, sort of. Although with this hot swap stuff it
really should be possible to add disks on the fly. That's the whole point.

Speaking of RAID controllers, we have a pair of identical R410's. And they BOTH
consistently produce these errors:

mptbase: ioc0: LogInfo(0x31123000): Originator={PL}, Code={Abort}, SubCode(0x3000)
mptbase: ioc0: LogInfo(0x31123000): Originator={PL}, Code={Abort}, SubCode(0x3000)
mptbase: ioc0: LogInfo(0x31123000): Originator={PL}, Code={Abort}, SubCode(0x3000)
mptbase: ioc0: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000)
mptbase: ioc0: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000)
mptbase: ioc0: LogInfo(0x31080000): Originator={PL}, Code={SATA NCQ Fail All Commands After Error}, SubCode(0x0000)

They produce these errors at a rate of around 10 per day throughout the day.
Both machines produce the exact same error. Same hex codes, etc. Identical. I
don't think it is actually a drive failing because the chances of both machines
failing at the same time in exactly the same way are slim. One of these
machines had what looked like a RAID controller crash which lost data and
didn't do our filesystems any good.

Whenever I call Dell tech support I always wonder why it is that Dell's phone
system always asks me for the long service code number instead of the shorter
service tag which is just the base-36 encoding (therefore much shorter) of the
service code. Sometimes I have one but not the other on hand. They are clearly
the same thing. Lots of people have even put up little webpages (which I have
used) that will convert from one to the other for you:

http://www.google.com/search?q=dell+service+tag+converter

Why would they ever ask for or deal in the long version and make me yell it at
them over a mobile phone in a noisy datacenter?

Then the next person I talk to wants the service tag again even though I just
told the phone system the service code.

Then the NEXT person wants to confirm the service tag.

At least they tend to understand the ICAO phonetic alphabet so we don't have to
haggle over whether I said b, c, d, e, g, p, t, v, z or 3.

I hate those pointless bezels that come with the machines. I try not to pay the
small amount of extra money for them anymore because they just go in a pile.
These machines sit in a datacenter, not a showroom.

Apparently there are at least two different kinds of DRAC: iDRAC Enterprise and
iDRAC Express. I suspect they are exactly the same hardware, perhaps with
different licensing or firmware.

My machines have iDRAC Express. iDRAC used to be something called BMC. Not sure
why they changed the name. The iDRAC stuff is nice. It took me a while to get
around to learning how to use it but it is worthy. Reminds me of some old
systems I had worked with in the past such as Sun, HP, and even Pyramid which
had service processors. I have long awaited the day that x86 servers got this
feature.

However, it has some weird limitations and is expensive compared to the latest
stuff from Supermicro. For example, it is odd that iDRAC Enterprise supports
public key auth and Express does not. The DRAC is a little processor (MIPS or
ARM on most platforms) running Linux or Busybox. Why not support public key? We
do everything with ssh keys. Without public key auth I have another password to
worry about.

A java applet for console in the DRAC web interface? With all of the 0-day
exploits for the JVM they want me to have the Java plugin enabled in my
browser? Why can't I just VNC? Or RDP? Tunnel it over ssl or ssh if you must.
The Java app is flakey. The JVM says "Downloading application" ...after a
couple of minutes that window will go away and be replaced by a window which
says "Unable to launch the application." It has "Ok" and "Details" as menu
options. If I click details it says "Error: Malformed reply from SOCKS server"
and a window full of XML. This happens sometimes. Restarting my browser
doesn't help. Hmm...I tunnel all of my web browsing through a SOCKS proxy with
SSH -D. I have an exception for the 10/8 network which doesn't get proxied.
Works for accessing the DRAC web interface itself. But the console java applet
is apparently somehow trying to use the proxy and failing. If I disable the
proxy in firefox the console applet works again. I would really rather just
VNC...

Once up and running he arrow keys don't work in DRAC java console. This is a
real problem in navigating BIOS and configuring things on, say, ESX console.
Turns out that you have to do some work to get them working:

http://ceph.github.com/sepia/drac/remote-console-keys/

http://www.anchor.com.au/blog/2011/03/evil-hack-to-make-arrow-and-sysreq-keys-work-with-a-dell-idrac-kvm-and-linux-desktop/

This last url says:

The KVM software makes a connection back to the iDRAC on the standard VNC
port (5900) (with the single use credentials that were provided to it in
the .jnlp file).

At this point, you could easily be mistaken into thinking, “Ah, VNC, that’s
got to work well right. Such a simple thing and all“. Unfortunately you
would be mightily wrong :( .

Whilst the iDRAC is using the standard VNC port, it appears that the
implementation has been somewhat customised.

So this is all based on VNC but Dell took standard VNC and fsckd with it! :(

All of the Dell DRAC SSL certs are the same with the same serial number. This
causes firefox to freak out and not accept it. Have to delete cert8.db from
firefox (stored cert cache) and restart firefox as a workaround.

The virtual media functionality in the DRAC doesn't seem to work properly in
Safari. I click virtual media and the little window where I can mount the media
never shows up. Works ok in Firefox/Linux.

Sometimes the DRAC web interface gets confused and all of the menu items become
labeled "undefined". Have to clear cache and try again and it works.

I've been using the Dell DRAC console quite a bit lately to remotely install OS
etc. It has terrible stuck-key/repeat problems. Typing slow helps but quite
often it is simply impossible to enter a 10 character password. Others have had
this problem:
http://lists.us.dell.com/pipermail/linux-poweredge/2008-January/034515.html
This is mostly a function of network latency and the fact the protocol sends
keydown/keyup messages. So if you get some latency longer than the interval
between your keydown/keyup the keyboard auto-repeat starts.

I added a Dell MD1120 disk array to an R415 with redundant external SAS
connections...BIOS complained:

Number of devices exceeded the maximum limit of the devices per quad
Please remove the extra drives and reboot system to avoid losing data
System has halted due to unsupported configuration.

Firmware upgrade fixed it. So again, it was broken when we bought it.

I was asked to evaluate the Dell H700 controller with Self-Encrypting Disks for
an organization with very serious security requirements. You key the controller
and then the controller keeps the key forever until manually cleared. So if
someone steals the server they can boot it right up and get the data. You can't
configure it to lose the key on power-off and require re-entry. I called Dell
and they took a couple of days to find the right people internally to ask and
confirmed that this is the case. Not very useful. This is only useful if the
server and the disks are separated.

I was trying to set up a new Dell R620. Got it racked, started to configure the
DRAC, couldn't ping it. Double checked everything, cabling, switch port, on the
right VLAN, went through all of the DRAC options. Noticed a little message
slyly hidden among the various config options: NIC Selection: Dedicated: A
require license is missing or expired. WTF? Dell sold us hardware (NIC on the
DRAC) which is completely useless without an additional license?

WHY!?!?!

My SuperMicro gear is SO much simpler. I've never upgraded BIOS or had IPMI or
RAID problems on any of them. It just works.

Cryptography Lessons

Tracy R Reed — Sun, 24 Jun 2012 01:00:00 -0700

The ever-excellent Khan Academy has produced a very nice and short series of videos explaining how cryptography works. Anyone who understands basic high school arithmetic can follow this. If you have ever been interested in the science of codes, ciphers, breaking them, etc. this is worth a look:

http://www.khanacademy.org/science/brit-cruise/cryptography

More in-depth treatments of cryptography can be found here:

https://www.coursera.org/course/crypto

and here:

http://www.youtube.com/playlist?list=PL71FE85723FD414D7&feature=plcp

And for the truly hard-core some of best books on crypto are:

http://www.amazon.com/Applied-Cryptography-Protocols-Algorithms-Edition/dp/0471117099/ref=sr_1_1?s=books&ie=UTF8&qid=1340524661&sr=1-1&keywords=applied+cryptography

and

http://www.amazon.com/Practical-Cryptography-Niels-Ferguson/dp/0471223573/ref=sr_1_1?s=books&ie=UTF8&qid=1340524712&sr=1-1&keywords=practical+cryptography

and

http://www.amazon.com/Cryptography-Engineering-Principles-Practical-Applications/dp/0470474246/ref=sr_1_2?s=books&ie=UTF8&qid=1340524751&sr=1-2&keywords=practical+cryptography

Federal Linux Security Resources/checklists

Tracy R Reed — Fri, 02 Mar 2012 15:35:00 -0800

I often say that most successful attacks and vulnerabilities are failures of imagination (when they aren't outright laziness/penny pinching). The authors of these documents have seen a lot of attacks and know something about how things should be configured to give your servers a fighting chance. These guides and checklists are great to look over for inspiration and ideas on how to better lock down your systems. Look over each item and think to yourself: "What on earth happened such that they had to put this on a security checklist?"

http://www.nsa.gov/ia/_files/os/redhat/rhel5-pamphlet-i731.pdf

http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf

http://people.redhat.com/sgrubb/files/hardening-rhel5.pdf (not Federal but good to review all the same)

http://benchmarks.cisecurity.org/tools2/linux/CIS_RHEL_5.0-5.1_Benchmark_v1.1.2.pdf (also not Fed but good)

http://web.nvd.nist.gov/view/ncp/repository

Use something like puppet to automate implementation of this stuff network-wide. That last NIST link even has an awesome puppet config for all of this! I've been reading through the code for the puppet modules and learned some neat things, including stuff I had no clue about previously such as how augeas works and what it is good for.

NIST, HIPPA, PCI, CIS, NSA, IQOQ, another day another security audit and industry-specific acronym!

Verisign intrusion

Tracy R Reed — Thu, 02 Feb 2012 14:24:46 -0800

Verisign has had some sort of intrusion, apparently: Key Internet operator VeriSign hit by hackers

Note that the Verisign CA business was sold to Symantec a couple of years ago (about when the attack happened) but it still operates under the Verisign brand. So who knows if the compromise is related to the CA in any way. You can bet Symantec is upset with Verisign over this because now their purchase of Verisign's CA business may have just lost value due to the branding. What if it was the CA network that was compromised and then sold to Symantec? That would really lead to some legal fireworks!

So far we have DigiNotar, Comodo, Realtek, JMicron on the list of compromised certificate authorities and each of them has been used to create bogus certificates. Hundreds of fraudulent yet CA-signed certificates happily accepted by browsers have been found in the wild impersonating websites/intercepting traffic and nobody knows how many more exist. Iran was successfully using bogus certificates signed for Google.com to intercept gmail and google chat traffic which has likely lead to deaths, given the nature of that regime and their attitude towards dissenters: http://googleonlinesecurity.blogspot.com/2011/08/update-on-attempted-man-in-middle.html

The web browser you are using trusts hundreds of different certificate authorities (any one of which could generate a certificate to impersonate any website they want or be compromise and used to do so) including CNNIC from China. I don't trust CNNIC any further than I could throw Mao Tse Tung's corpulent carcass.

Verisign is a big company which provides many services and no doubt extensively subnets and divides up their networks as required by PCI among many other security standards. One would hope, for example, that the corporate office network (a very common way to infiltrate a network) is in no way connected to the DNS or CA infrastructure (now with Symantec but there could still be links) so that an intrusion in one of these areas would not affect the rest. I find these two paragraphs the most disturbing:

  The VeriSign attacks were revealed in a quarterly U.S. Securities and Exchange
  Commission filing in October that followed new guidelines on reporting
  security breaches to investors. It was the most striking disclosure to emerge
  in a review by Reuters of more than 2,000 documents mentioning breach risks
  since the SEC guidance was published.
  ...
  Ken Silva, who was VeriSign's chief technology officer for three years until
  November 2010, said he had not learned of the intrusion until contacted by
  Reuters. Given the time elapsed since the attack and the vague language in
  the SEC filing, he said VeriSign "probably can't draw an accurate assessment"
  of the damage.

The attacks were revealed only to the degree legally required by the SEC and buried in a quarterly 10-Q filing in the hope everyone would overlook it. The CTO wasn't informed (or isn't admitting to having been informed) and the whole thing was brushed under the rug for two years. That's way sleazy.

What does it mean for us? Probably not much, at least at first. If people understood how the CA system worked Verisign's brand would be affected and people would put less trust in their certificates and be less likely to input their credit card number. While it is the part of the system most people focus on, we don't pay a CA to encrypt our traffic. We can do that without them. We pay them to certify that our server is who it says it is. If the media were to run with the idea that the CA system is broken and untrustworthy (which it is) and that man-in-the-middle attacks are rampant (they happen but aren't common, relatively speaking) it could really hurt the e-commerce industry in general which would be bad for us.

PA-23 Piper Aztec checkout

Tracy R Reed — Sat, 27 Aug 2011 12:55:00 -0700

You can check it out here.

Comments on "Views regarding PCI compliance are mostly positive"

Tracy R Reed — Wed, 12 Jan 2011 18:00:42 -0800

A few comments and observations regarding Views regarding PCI compliance are mostly positive

Most IT security practitioners believe...

Sure, they are the ones for whom work is being generated by PCI! :) A survey of business owners/CEOs would be much more interesting.

A majority of survey respondents were "very confident" they could
pass an assessment today.

Hmm....these respondents need to read the Verizon 2010 Payment Card Industry Compliance Report: http://www.verizonbusiness.com/go/pcireport where 78% were non-compliant at IROC. My experience also is that organizations are not nearly as compliant as they think they are. They tend to make assumptions without actually reading the requirements.

The card brand, however, reports only "moderate" compliance for smaller retailers.

I bet that is putting it mildly!

"The people and education is a big issue that maybe is more
challenging to address than just putting a technology in place,” Kost
said.

Definitely! We were just discussing this very issue on here...attitudes are hard to change.

Needing to upgrade antiquated systems to bring them into
compliance is the second greatest pain point...

Ditto again. I have a client running Fedora Core 3 systems (not necessarily in the CDE) with so much stuff all on one system (against best practice and a violation of PCI if in scope) that it has been nearly two years and we still have not been able to move/upgrade it!

Dan Langin, a Kansas lawyer who advises clients on PCI
compliance, told SCMagazineUS.com on Wednesday that organizations
commonly have challenges with the step that requires they maintain a
policy that addresses information security.

I'll have to remember that name. Never before ran into a lawyer who specializes in PCI.

This requirement is somewhat objective and it can be difficult to
determine whether the organization is actually in compliance, he
said.

And I think he means subjective, not objective... And this is the point where one of my clients is currently stuck. They have most of the technical requirements met but need to do some documentation and education of policies.

The cost to achieve PCI compliance is often tied to an
organization's size, with larger companies spending more than their
smaller counterparts, Kost said. Sixty-two percent of all respondents
said they have spent at least $100,000 on compliance over the past
five years.

It is also tied to "technical debt". If you have a very messy environment with interdependencies all over the place running on systems which have been EOL for ages it is going to cost a whole lot more. Such is the case with the Fedora Core 3 client above.

Most organizations plan to increase PCI compliance spending in
2011, with some organizations planning to invest in technologies that
allow them to comply in virtualized environments, according to the
survey.

What sort of technologies would they need to allow themt o comply in virtualized environments? The virtualization container is to be secured to at least that of the highest level virtual machine running in it.

Meanwhile, 60 percent of respondents said they are using another
emerging technology – point-to-point encryption (P2PE), sometimes
referred to as end-to-end encryption

I would hardly call P2PE an merging technology. VPN, SSL, etc. which we have had for many years now are P2PE. I think the innovation here is in having it built into a PED.

SIP brute force attacks

Tracy R Reed — Sun, 26 Dec 2010 23:36:06 -0800

If you run a VOIP system accessible to the Internet you need to keep up on your system security. Over the last year I have seen an ever increasing amount of brute force attacks on SIP servers. Many systems have poorly chosen passwords which are being discovered via brute force guessing. The bad guys then route international phone calls to their phone company or calling card service through your phone system leaving you stuck with a huge bill. I personally knew someone who ended up with a $100k+ phone bill on his DS-3 line. VOIP providers are now making people sign contracts accepting all responsibility for charges incurred if the customer's VOIP system is breached.

Checking my own system logs I see 137,507 failed SIP REGISTER attempts over just the last 4 days.

In addition to choosing strong passwords (and why not, it isn't like human beings are going to be using these passwords, you program them into the phone) the only real solution (more of a band-aid, but a really good one) is to use a package like fail2ban (in the rpmforge repository if you use CentOS/Fedora as well as the standard apt-get repo for Debian/Ubuntu) to block too many failed register attempts in a row or to use a set of iptables rules like the following which I cooked up a long time ago and have used to remediate DOS attacks on HTTP, ssh brute forcing, and now SIP brute forcing. This does not detect failed registers but merely checks how many times a remote host sends a UDP packet to 5060 per amount of time. You may have to play with these numbers depending on how chatty your phones are. And be sure to whitelist generously. The chances of you whitelisting someone who will attack you are very small. The chances of locking out a legit client are decent. If someone can't register and you can't see their register attempts on the console check your packet filter. But it is worth it. Also keep an eye on your logs using some sort of log monitoring software (I wrote my own, used to have a website for it but not anymore, need to get it back up there) so you can know about failed/blocked registrations.

# Deal with SIP brute forcing
iptables -N SIP_WHITELIST
# home
iptables -A SIP_WHITELIST -s 1.2.3.0/24 -m recent --remove --name SIP -j ACCEPT
# voip provider
iptables -A SIP_WHITELIST -s 4.5.6.0/24 -m recent --remove --name SIP -j ACCEPT
# remote location
iptables -A SIP_WHITELIST -s 7.8.9.0/24 -m recent --remove --name SIP -j ACCEPT

iptables -N SIP_BRUTEFORCE
iptables -A SIP_BRUTEFORCE -m recent --set --name SIP
iptables -A SIP_BRUTEFORCE -p udp --dport 5060 -m state --state NEW -j SIP_WHITELIST
iptables -A SIP_BRUTEFORCE -m recent --update --seconds 30 --hitcount 3 --name SIP -j LOG
iptables -A SIP_BRUTEFORCE -m recent --update --seconds 30 --hitcount 3 --name SIP -j DROP

iptables -A INPUT -p udp --dport 5060 -m state --state NEW -j SIP_BRUTEFORCE

You cannot rely on antivirus

Tracy R Reed — Fri, 19 Nov 2010 14:29:20 -0800

As the number of viruses/malware to scan for and parts of the system to monitor for infection increases more resources will be required. I bet we already spend the equivalent of one whole CPU of ten years ago just scanning for malware on the typical modern computer. In the last couple of years there has been talk of the end of antivirus as we know it:

http://www.google.com/search?q=the+end+of+antivirus

This is because the increase in resources required to secure the computer cannot continue forever.

What's worse is that antivirus only detects known viruses. There are an increasing number of unknown viruses out there and antivirus vendors are falling further behind. There has always been a large lag between initial release and detection by antivirus software. The virus has to be released, discovered, reverse engineered, signature created, added to antivirus software signature database, then the user has to update. This all takes plenty of time.

There is an increasing number of unknown viruses out there that do not get caught until after they have already caused damage. Google had no clue they were infiltrated until the bad guys tipped their hands by getting caught logging into other people's webmail accounts which prompted investigation. At that point the malware they had been sent was undetected. There have been serious consequences, likely including prison time if not worse, for certain human rights activists in China whose gmail accounts were compromised.

Stuxnet was discovered in June 2010. The widely accepted theory is that it was designed to sabotage the Iranian centrifuges and has probably been successful. According to:

http://www.fas.org/blog/ssp/wp-content/uploads/NumberCentrifuges1.jpg

we see the number of centrifuges online decreasing between May and August of 2009.

On July 17, 2009 WikiLeaks posted a notice saying:

Two weeks ago, a source associated with Iran’s nuclear program
confidentially told WikiLeaks of a serious, recent, nuclear accident at
Natanz. Natanz is the primary location of Iran’s nuclear enrichment
program. WikiLeaks had reason to believe the source was credible however
contact with this source was lost. WikiLeaks would not normally mention
such an incident without additional confirmation, however according to
Iranian media and the BBC, today the head of Iran’s Atomic Energy
Organization, Gholam Reza Aghazadeh, has resigned under mysterious
circumstances. According to these reports, the resignation was tendered
around 20 days ago."

A centrifuge full of uranium hexaflouride turning at 15k RPM failing and spewing its contents widely throughout the facility due to someone messing with the speed controls via the computer which controls the PLCs is indeed a serious nuclear accident which could end the career of whoever is in charge.

All of this implies that it was more than 10 months that Stuxnet was out there completely undetected by antivirus.

What malware is on the computer you read this on that you won't know about for 10 months?

Windows market share

Tracy R Reed — Sun, 10 Oct 2010 20:40:21 -0700

http://www.zdnet.co.uk/blogs/the-open-source-revolution-10014902/microsofts-dwindling-market-share-10020700/

It is debateable whether Linux is increasing in desktop market share or not although it is killing (in the form of Android, for whatever that's worth) in the mobile phone business. I think Linux probably is increasing in desktop market share or perhaps number of installations if not increasing as a percentage because I have come across more and more people in the last few years who are using it. I recently saw someone post on Facebook (who I have not influenced) that they haven't had any computer virus problems since they have been using Linux.

I have been reading this blog of the HeliOS Project for a long time:

http://linuxlock.blogspot.com/

Thanks to Ken Starks there are thousands of Linux computers out there in Texas homes (now doing 300-400 per year and doing it since 2002) helping less fortunate kids get an education. Not only has he endured technical and political challenges but note where he mentions he took a stabbing in the course of trying to deliver a computer to some kids. That's dedication!

And obviously Mac has gained market share in recent years which is surely where most of the Windows share is going.

As markets mature they have historically had a tendency to standardize on open standards and I think we are finally beginning to see that in the computer business. Decades ago the hardware and software were both completely proprietary. Then the hardware became somewhat standard with the introduction of the PC as a platform as IBM hardware went by the wayside. Then networks were standardized on ethernet and TCP/IP and the various proprietary networking protocols and hardware went away. And now we see the beginning of serious change affecting the average user in the software side of things. It's a good day.

Aeronautical charts

Tracy R Reed — Thu, 23 Sep 2010 13:10:00 -0700

The special aeronautical charts (aka maps) used by pilots have expiration dates. Dates vary between every 56 days to every two years depending on the chart. Mountains don't generally change much but antenna towers, roads, and sometimes even airports and towns do. Sometimes the radio frequencies for control towers change or airspace boundaries move. So the charts get updated.

When my charts expire I have to buy new ones. That means at least every couple of months I am having to buy something new. Most pilots just throw away the old charts. But they are so densely packed with cool information that I hate to just throw them in the trash. I recently noticed that I had a stack of expired charts about a foot high and wondered what I should do with them.

When I was a little kid I liked playing with Flight Simulator on the computer. Someone gave me a copy of an aeronautical chart to use with the flight simulator software and as a Junior High School kid I was making simulated cross country flights. By the time I started training to be a "real" pilot I already knew most of what I needed to read a chart and navigate. I think it would be cool to help out some other kids with their interest in aviation (sort of like paying back the good karma that came to me when I was a kid). So I bundled up the charts into useful sets (typically a Los Angeles sectional, Los Angeles Terminal, San Diego Terminal, IFR enroute covering SoCal, and book of instrument approach procedures covering SoCal per set). Then I placed an ad in the Free section of the San Diego Craig's List. I really don't know many kids who play with the flight sims anymore and it isn't generally considered nearly as cool as it once was to be a pilot so I wasn't sure if anyone would be interested.

Over the next few hours my inbox was full of people willing to drive across town to pick up a set. Mostly for their own kids, as well as a couple inactive pilots looking for study materials to get back into it, and one youth pastor looking for something cool to put up on the walls of his classroom.

All of the chart sets are now spoken for and given away. Hopefully there will be a lot of happy kids out there this evening.

PHP

Tracy R Reed — Sat, 22 May 2010 04:25:00 -0700

It has been a while since I did any language trolling so let's talk about a religious issue: PHP. You either love it or you hate it. I have been looking at PHP frameworks lately. Not that I want to but in this particular case there isn't much choice. I am constantly reminded of all of the reasons why I played with PHP for 6 months then moved away from it so many years ago and have since watched the train wreck from a safe distance.

PHP has no language design philosophy.

With Ruby/Java/Python (almost) everything is an object. Perl has its "There's more than one way to do it" swiss army chainsaw. Python has "There's one obviously right way to do it" and zen of python in the "import this" easter egg. Lisp/Haskell/Erlang all have their (purely)functional related philosophies. PHP? Nada.

Not a general purpose language.

Really only meant for webpages. Not a general programming language. While technically you can run php from the command line and automate system tasks with it few people actually use it that way with shell, perl, python preferred instead. I have never downloaded a command line app to find that it is written in PHP. Being able to use the same language for web programming and, say, system automation is a handy way to amortize that learning curve over a number of projects.

Code embedded in HTML is bad

Originally started out as a way to embed basic site counters and such in HTML. It took years but a few people eventually figured out templates were the way to go and started heading towards MVC with templates etc. But far too many people still write PHP intertwined with HTML making a real mess.

Breaks with Apache WebDAV

I set a developer up with WebDAV access to a PHP project so he could edit code, upload files, etc. The very things that WebDAV was designed to do. But whenever he would open up a .php file the editor would come up empty. It turns out that Apache was trying to EXECUTE the PHP file instead of just serving it up for the developer to edit. I found a few references to how to disable the PHP engine when a file is being served up via WebDAV but have so far been unsuccessful in implementing the suggested fix.

Terrible security history

Long a major complaint against PHP and very sensitive subject to PHP programmers. Before flaming me go patch your Wordpress instance. You know it needs it.

The infamous register globals functionality is where people always start when complaining about PHP security. It took years but most people have moved away from using them, especially after it was turned off by default in 4.2. But why did anyone ever think it was a good idea in the first place?

I saw quite a few php apps get owned by the xml-rpc flaw in the 2005-2007 timeframe.

In 2007 it was discovered that you could insert PHP code into a gif name it image.gif.php and get it executed on many sites. Oops. Another instance where parsing your code out of content being served up (instead of separate of logic and presentation via templates) was a big mistake.

I am yet to see anyone religiously use prepared (aka parameterized) statements in PHP to prevent SQL injection which consistently leads to total box ownage. The attackers are generally far more creative than the programmers/admins and find ingenious yet obvious in hindsight ways to pull this off. No, addslashes() is NOT sufficient. mysql_real_escape() is not sufficient either. Take a look at this blog entry from 2007 which sums up some of the more popular options: http://old.justinshattuck.com/2007/01/18/mysql-injection-cheat-sheet/ Scroll down to where it mentions BIG5 and advanced injection techniques. It is a fact that there is no amount of clever escaping which will suffice. Parameterized queries are the ONLY solution to this problem. Parameterized queries simply makes it impossible to get user input code parsed as part of the SQL statement.

While you can code sql injections in almost any language that speaks SQL to an RDBMS many other languages/frameworks do it differently by default. PHP's default database interface seems to encourage SQL-injections. The magic quotes hack is just an ugly band-aid. Same for addslashes(). Compare that to Java where you have Command and Parameter objects or can abstract away the SQL generation with Hibernate. Or compare it to Python/Django which also has an ORM which uses parameterized queries. Or to Ruby with its Rails ORM and parameterized queries. Or Zope/Plone with its object database and the inherent impossibility of SQL injection no matter what the programmer does.

And who can forget all of the file inclusion vulnerabilities? Remote file inclusion even! There have even been instances of code injection into PHP regexes via null bytes.

PHP wildly mixes code with strings all over the place causing this mess.

PHPs promiscuous mixing executable code with web content causes trouble as well. 1% of sites accidentally reveal their db passwords: http://www.feross.org/cmsploit/

The latest trend is exploitation of memory corruption bugs in the PHP interpreter itself. You might think that because you are using an interpreted language you don't have to worry about pointer mishandling, buffer overflows, etc. But it just isn't so. And PHP has such issues. There are right now undisclosed remote exploits which have not been revealed to the public and have not been patched in the PHP interpreter. There will be a talk on this given at SyScan Singapore 2010.

And then there is the issue demonstrated by this video: http://www.youtube.com/watch?v=6W68u18Bh28&NR=1 PHP apps traditionally mix static content and PHP code in the same filespace. So if you can get your own PHP code uploaded you can get it executed and suddenly you are owned.

You can say that all of this security attention is due to simply being a very popular language for implementing webapps if you like (although popularity leading to exploits is one of my favorite myths to bust) but it doesn't change the fact that there are a lot of problems which either don't exist or are simply less likely to be exploited in other languages/frameworks.

Be sure to keep an eye on http://php-security.org at least to know what you are up against.

PHP is almost everyone's first web programming language

This may hurt the feelings of a lot of PHP programmers whose business cards say "Software Engineer" but it is an important part of the argument.

Far too many people pick up a PHP tutorial, start coding knowing nothing about software development best practices or security issues, and turn loose some code on the net. This has nothing to do with the language necessarily but a lot to do with the community surrounding it. They are typically either newbies or at the very least not nearly as wise as they think they are.

If PHP is the only web programming language you have much experience with you have a problem.

ReST? MVC? Unit tests? What are those? I know that there are a few PHP programmers out there who practice these things but I have never met one personally.

Too many people start with PHP but then never graduate to any of the other languages/frameworks.

The old "because it's the most popular" myth

Tracy R Reed — Fri, 02 Apr 2010 23:47:05 -0700

Linux is a very tempting target for spammers and botnet owners. And there are millions of Linux boxes out there by now. But so far the only way they are really being compromised is through PHP web apps and poorly chosen passwords. Linux machines are being constantly bombarded with ssh brute force attacks and funny url requests. And as I manage my ssh access well and don't run publically accessible PHP apps I don't have problems. Between the MySQL on Windows worm a few years ago:

http://dev.mysql.com/tech-resources/articles/security_alert.html

and the Linux on MIPS router exploit from last year:

http://blogs.zdnet.com/security/?p=2972

and the Apache on FreeBSD worm:

http://news.cnet.com/2100-1001-940585.html?tag=fd_top

and the recent Linux router based botnet:

http://www.computerworld.com/s/article/9159758/Chuck_Norris_botnet_karate_chops_routers_hard

they are clearly trying anything that is exploitable including the very obscure software platforms. I just don't buy the idea that they only go after Windows because it is the most common. That is just where the low hanging fruit is and has the most exploits.

Software design has got to have something to do with it and being forced to maintain decades of backwards compatibility and poor design decisions as part of holding onto their monopoly has got to complicate things for Microsoft.

I actually like reading about Linux based appliances with poor security defaults being attacked. It really shoots down the whole idea that only Windows is targeted and that this is because it is the most popular. Notice that the primary way in which Linux systems are being attacked is misconfiguration or poor choice of password. Both are easily remedied issues. Actual exploitable implementation flaws are more rare than in Windows and actual design flaws rarer still.

Flight to Catalina Island (KAVX)

Tracy R Reed — Fri, 26 Mar 2010 13:00:00 -0700

On March 21st, 2009 (Yes, I'm a little behind in my blog entries!) I flew my wife and three of her friends to Catalina Island in Plus One's Cessna 210 N210BX. Catalina Island is one of the "Channel Islands" about 30nm off the coast of Los Angeles. We departed from Montgomery Field (KMYF) in San Diego which is 76nm away from the island. This is usually about a 40 minute trip since I like to climb up high going out over the ocean. This is a fun place to fly to for various reasons. I have flown out there probably 20 times by now.

From a pilot's point of view the trip out over the ocean is something you don't get to do often unless you fly the international big iron. There's something cool about seeing only water in any direction. Aside from my many Catalina crossings, the only other time I have flown across a long stretch of open water is when I had to fly due west across the Sea of Cortez from Culiacan to La Paz. There was a tropical storm to the north of Culiacan on my way to San Diego from a friend's place south of Puerto Vallarta.

Originally constructed in the late 1930's, the Catalina airport itself is on top of a mountain. Each end of the runway is practically a sheer cliff. The runway has a hump in the middle so that when touching down (or departing) you can't see the other end. Rumour has it that pilots have been known to think the peak in the middle of the runway is actually the end of the runway and slam on the breaks or initiate a go-around. Since it is often impossible to tell if someone is departing the runway going the opposite direction it is very important to use the UNICOM (local airport radio communications frequency) to be aware of what is going on and announce your intentions. The field has no official tower or controller but there is a tower of sorts above the terminal building where you go to pay landing fees and book transportation down to the town. There is usually an employee in there monitoring the UNICOM who will announce winds and help out within their abilities. The winds around the island can be tricky as you can get up and down drafts right around the cliffs on each end of the runway.

And then there is the fact that the airport itself is at 1,600' MSL (Mean Sea Level) elevation. And that is what made this day's trip more interesting.

We got a rather late start due to the low marine layer clouds that often cover the coast. I am a current instrument rated pilot and our 210 is a capable airplane with IFR instrumentation and a Garmin 530 so normally a marine layer is no problem. But if the clouds start at 1,900' MSL as they did on this day and Catalina Airport is already all the way up at 1,600' MSL that means there is only 300' of clearance between the runway and the bottom of the clouds. That is not enough room to safely get there and maneuver to a landing.

The combination of warm landmass and/or a light breeze often produces a bubble of higher ceilings over the island. On this day I met a pilot who had flown at about 1000' MSL the whole way over all that ocean (from LA but I know people have done this from San Diego too) and then quickly climb as they approach the cliff-face at the end of runway 22 at Catalina entering that bubble of higher ceilings immediately over the island just barely clearing the cliff making it up to runway level and then plop it right down on the deck. But that's not for me. Nor do I recommend it for anyone else. If you have a problem at 1000' over the ocean you have little time and even fewer options. Not only does this risky maneuver likely violate VFR (Visual Flight Rules) cloud clearance requirements but it leaves too few options should anything not go exactly as planned. Any go-around is likely to involve going into IMC (Instrument Meteorological Conditions). I insist on a normal, stabilized approach to landing. Lack of pilots choosing to go around has cost the club some bent aircraft in recent years. It is never a good idea to do anything which would preclude the ability to go around. Recall that the number one cause of weather related general aviation accidents is VFR into IMC.

So we waited. Some of us more patiently than others. A pilot must resist get-there-itis, especially when it comes from passengers, even if that passenger is the pilot's wife. Eventually the weather reported that the ceiling was 500' above AVX which put it at 2100' above sea level. Departing Montgomery Field (KMYF) in San Diego with a VFR-on-top instrument clearance to OCN (Oceanside) VOR (Variable Omni-range, a navigation beacon on the ground) we climbed up through the clouds, canceled our IFR clearance upon reaching clear skies, and then on up to 8,500' for the cruise out there. I was hoping that things would begin to clear during the flight to the island. As the surface of the island warms it will often burn a hole through the marine layer and sometimes you will find the island sitting in the clear surrounded by clouds. I knew this was unlikely to happen on this day as the temperature was just too cool. But I had a plan B and plan C.

We agreed before take-off that when we got to Catalina Island if there was no way to get down in clear skies I would attempt the VOR instrument approach to landing. That was plan B. And if that didn't get us down into clear view of the airport we would execute the missed approach procedure, get back on top, and then we would fly about an hour to the east and spend the day in Palm Springs instead of Catalina. This was plan C.

I have always considered the VOR approach to Catalina Island to be a fairly useless approach and never expected it would really get anyone below a marine layer. The airport is at 1,600' MSL. With this instrument approach you can get down to 2,440' MSL over the airport. This means you need at least 840' between the clouds and the runway. The marine layer is usually lower than that. When we departed it was reported that there were 500' ceilings.

Having descended from cruise altitude down to around 4,500' and approaching where my calculations told me the island should be and seeing nothing but clouds I advised SoCal approach that I would need an IFR clearance for the VOR-A approach to Catalina while beginning to slow the airplane from cruise speed to approach speed. They cleared us for the approach and with the missed approach procedure in mind and ready to execute we passed over SXC VOR nearing 90kts and tear-dropped into the holding pattern for a turn for alignment with the approach and started a descent down to 2900' which plunged us down into the clouds. There are only 1.6 nautical miles between the FAF (Final Approach Fix) to the MAP (Missed Approach Point) with an MDA (Minimum Descent Altitude) of 2440'. If you have the airplane slowed down to 90knots for the approach you have one minute and four seconds to descend from 2900' to 2440' which means you have to descend at 431 feet per minute to reach the MDA on time. If you go faster you must descend faster and have a smaller margin for error.

Upon passing the SXC VOR (which marks the FAF) inbound we turned to heading 352 degrees while keeping one eye on the time (counting down 1m and 4s), one eye on DME (Distance Measuring Equipment, to tell us when we are 1.6nm from the Catalina VOR on a mountaintop nearby the airport as a cross-check to the time), one eye on the attitude indicator (to keep us right-side up inside the clouds), one eye on the airspeed indicator (trying to maintain 90knots to make all the math work out correctly) and one eye on the compass trying to maintain 352 degrees. You didn't know instrument pilots have 5 eyes? They do. And at least as many hands.

Just as we passed through 2500' MSL we could see the ground. A few seconds later we were at 2440 and the airport had come into view off to our right. Ideally we would have come out right above it. With only 1.6 miles you don't have much room to get lined up on your outbound radial or established on your compass heading and we actually ended up passing just slightly north of the VOR on our way inbound according to the GPS which I suspect is what did it. We made a right turn into the downwind leg of the pattern while simultaneously calling SoCal to cancel our IFR clearance since landing was assured, announced our presence to any other local traffic on the UNICOM frequency (no control tower at this airport), ran a GUMPS (pre-landing) check one last time (landing gear had already gone down at the FAF), made a couple more turns in the pattern and gently squeaked the wheels onto the pavement. Mission accomplished! Apparently, my wife had been doubting our ability to land when we arrived to find the island cloud covered. She excitedly pointed out the airport when it appeared and upon exiting the aircraft I was promptly declared her “hero”!

Unfortunately, it was now around noon. The airport would close at 5pm after which no more takeoffs would be allowed. While open to the public this is actually a privately owned airport and has somewhat restricted hours. We planned to go back that same day. After landing I paid the $25 landing fee and then bought the five of us $25 round-trip van tickets for a 30 minute van ride down the mountain to the coastal town of Avalon, the only town on the island.

We had three hours to look around. It's a small place and you can walk from one end of the main drag to the other in 15 minutes. But the ladies spent a lot of time in each little gift shop along the way. We walked around town and ate buffalo burgers and oysters at a local burger joint with some sort of tropical island theme whose name I don't recall. I've eaten at nearly every restaurant on the island it seems. Many pictures were taken. By the time we had lunch and made it from the docks on one end to the historic "Casino" (not a place of gambling, simply entertainment) on the other it was time to head home. As usual, we got a pretty good look at some buffalo along the winding road from the airport to town and on the way back up. The island was the greenest I have ever seen it due to the recent rains. I also saw a number of scorched tree trunks from the wildfires they have had there in the last couple years.

At 4pm we met the van for the ride back up the mountain to the airport. The van left a few minutes late and we stopped to look at some buffalo on the way up. So we had around 20 minutes to get off the ground. While the passengers made final bathroom breaks and got themselves situated in the plane I did the pre-flight. Then hop in for the start checklist, start, taxi, final takeoff checks, and we were off the ground with only a couple minutes to spare. It was tight but we made it.

After takeoff we were still underneath the marine layer although it was higher now than when we arrived. A few miles from AVX I called SoCal for an instrument clearance to Montgomery so we could get above the clouds for a safe open-water cruising altitude. This was quickly granted and up we went through the clouds. The rest of the the flight back to Montgomery was uneventful aside from nice scenery. The clouds had mostly cleared by the time we got back although I stayed on the IFR flight plan and flew the ILS (Instrument Landing System) into MYF for a smooth landing and happy conclusion to a successful day-trip to Catalina Island.

Change and HTML emails

Tracy R Reed — Tue, 09 Jun 2009 11:30:00 -0700

I write a lot of lengthy emails, reports, and other documents and all
too often forget to post them here for others to enjoy (or ignore or
despise, whatever). I know IT people who are staunchly against change,
usually old guys. Maybe they are jaded and burned too many times by
change or maybe they just don't want to learn something new and are troubled
by watching their skillset slowly become obsolete. And I also know guys
who are always chasing the latest and greatest but not really getting much
productivity out of it. I am always looking for that optimal middle-ground...

On Tue, Jun 09, 2009 at 01:29:56AM -0700, Raleigh spake thusly:
> In fact, I've always been curious why so many IT pros that I meet
> are anti-change when it comes to software. Aren't we supposed to be
> the technology evangelists within our respective organizations? It
> is the job of our users to be cryin' about change. Not us.

I run into this often. Some people accuse me of always chasing the
shiny and some people accuse me of being an old fart who won't
change. I'm against change simply for the sake of change. Have there
really been any major breakthroughs in UI research in the last 10
years? Not really. So why are the UIs in certain products changing so
much? For the same reason car bodies change every year: Marketing and
change for the sake of change. I'm not into that. It creates
difficulties in training, introduces new bugs, and doesn't really
benefit us, the end users.

Virtualization, on the other hand, is a huge breakthrough on the x86
platform (IBM big-iron has had it for decades) and that is change that
I can definitely get behind and advocate constantly.

> P.S. - speaking of change, when will the list software (whatever
> that is) that this list runs on be upgraded to allow rich text or
> HTML email. I'm subscribed to various lists (IT and non-IT related),
> and this one is the only one that is plain text only. It's 2009, for
> crying out loud. Time to get with the 21st century. Just a
> thought...

This is more change for the sake of change. Plaintext works great for
what this mailing list is intended for. There are many exploits and
browser compatibility issues (in all of HTML, CSS, and Javascript) and
I don't want random strangers mailing active content into my
mailbox. Phishing would be largely impossible if it weren't for HTML
email. I read email on an OS that is definitely not prone to these
problems and I *still* avoid non-plaintext emails. I also don't want
your emails with the kitty backgrounds and yellow on black text and
animated corporate logo gifs in signatures etc. You got something to
say? You can say it just fine in plaintext. Impress me with your
beautiful prose not your idea of beautiful (which I will probably
consider gaudy) artwork.

--
Tracy Reed
http://tracyreed.org

I'm on twitter

Tracy R Reed — Mon, 06 Apr 2009 21:30:00 -0700

By popular demand you can now all know what I'm having for lunch in real time. Isn't technology great?

http://twitter.com/tracyrreed