[iwar] [fc:CIOs.band.to.stop.paying.for.software.bugs]

From: Fred Cohen (fc@all.net)
Date: 2001-10-16 15:31:07


Return-Path: <sentto-279987-3022-1003271478-fc=all.net@returns.onelist.com>
Delivered-To: fc@all.net
Received: from 204.181.12.215 by localhost with POP3 (fetchmail-5.1.0) for fc@localhost (single-drop); Tue, 16 Oct 2001 15:32:07 -0700 (PDT)
Received: (qmail 23060 invoked by uid 510); 16 Oct 2001 22:30:58 -0000
Received: from n18.groups.yahoo.com (216.115.96.68) by 204.181.12.215 with SMTP; 16 Oct 2001 22:30:58 -0000
X-eGroups-Return: sentto-279987-3022-1003271478-fc=all.net@returns.onelist.com
Received: from [10.1.1.223] by n18.groups.yahoo.com with NNFMP; 16 Oct 2001 22:31:18 -0000
X-Sender: fc@big.all.net
X-Apparently-To: iwar@onelist.com
Received: (EGP: mail-8_0_0_1); 16 Oct 2001 22:31:18 -0000
Received: (qmail 8951 invoked from network); 16 Oct 2001 22:31:10 -0000
Received: from unknown (10.1.10.26) by 10.1.1.223 with QMQP; 16 Oct 2001 22:31:10 -0000
Received: from unknown (HELO big.all.net) (65.0.156.78) by mta1 with SMTP; 16 Oct 2001 22:31:08 -0000
Received: (from fc@localhost) by big.all.net (8.9.3/8.7.3) id PAA24039 for iwar@onelist.com; Tue, 16 Oct 2001 15:31:07 -0700
Message-Id: <200110162231.PAA24039@big.all.net>
To: iwar@onelist.com (Information Warfare Mailing List)
Organization: I'm not allowed to say
X-Mailer: don't even ask
X-Mailer: ELM [version 2.5 PL1]
From: Fred Cohen <fc@all.net>
X-Yahoo-Profile: fcallnet
Mailing-List: list iwar@yahoogroups.com; contact iwar-owner@yahoogroups.com
Delivered-To: mailing list iwar@yahoogroups.com
Precedence: bulk
List-Unsubscribe: <mailto:iwar-unsubscribe@yahoogroups.com>
Date: Tue, 16 Oct 2001 15:31:07 -0700 (PDT)
Reply-To: iwar@yahoogroups.com
Subject: [iwar] [fc:CIOs.band.to.stop.paying.for.software.bugs]
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 8bit

<a href="http://www.cio.com/archive/101501/wasting.html">http://www.cio.com/archive/101501/wasting.html>

Greg Seyk, newly appointed CIO of VisionQuest, had only months to rid his
organization of its Y2K bugs. (Y2K, after all, was not a top priority for
the Tucson, Ariz.-based national youth services organization, whose mission
is to keep kids out of jail, and Seyk had just come on board the previous
spring.) 

The first thing Seyk did was purchase a Y2K-compliant ERP system‹Lawson
Insight version 7.1.5‹from Lawson Software, a St. Paul, Minn.-based vendor.
But just as he and his staff of five got around to testing the software,
Lawson released a new version (Lawson Insight version 7.1.6) that included a
function for prioritizing bill payments. That function had been promised but
never delivered in any of the previous versions, Seyk says.

With just two months to go before the clock struck midnight on Dec. 31,
1999, Seyk didn't have time to deploy the upgrade, even though the payment
prioritization function had been a critical selling point for the
53-year-old CIO. "We had to implement accounts payable, the general ledger,
payroll and human resources to make sure they were Y2K compliant. It was no
small feat," says Seyk, who is also a vice president of the private company.

Between November 1999 and July 2001, Lawson released seven new versions of
its software to fix bugs or add functionality that had been promised but
absent in each previous version. Seyk was outraged. He documented his
problems in a series of letters to Lawson executives, met with them on two
occasions, and sank a total of $594,974 into software and maintenance to
correct the flaws in their products.

And then it dawned on Seyk why the software and support were so bad: That's
the way vendors make money. They push products on the market before they've
been adequately tested, demand payment up front and then are often not
available to deal with the sequelae of poorly performing products. (Lawson
officials declined to comment specifically on VisionQuest's problems with
its software. All a Lawson spokesperson would say is that the company is
working with VisionQuest in an effort to resolve its concerns. "We are
committed to 100 percent customer satisfaction," says Bev Bergstrom, vice
president of communications for Lawson.)

How Bad Software Pays Dividends
CIOs have been complaining about poorly designed and buggy software forever.
In a recent survey on CIO.com, almost half of the 88 IT professionals
questioned said they were unsatisfied with both the quality of their
business software and the support. (For full survey results, see Does
Business Software Stink?) The problem is a big one: Faulty software costs
businesses $78 billion per year, according to Jim Johnson, chairman of The
Standish Group, a research company based in West Yarmouth, Mass.

But now many CIOs are beginning to realize that the root of the problem may
lie in the economics of the industry. Vendors generate most of their
revenues through perpetual licensing agreements, which force CIOs to pay up
front for an application. In return, CIOs own the software and the right to
use it "in perpetuity." The problem with this model is that in reality, CIOs
are lucky if they can get three years out of a product before vendors
release entirely new versions of their software. Vendors further pressure
CIOs to buy those new releases by threatening to stop supporting previous
releases‹a tactic they often take both to cut their tech support costs and
to get CIOs to pay again and again for what is essentially the same product.

Another problem with the perpetual model is that CIOs have
 
"We didn't pay Oracle. We owed maintenance of $300,000 to $400,000, and we
just didn't pay it. We said, 'We're holding on to the money until you get
this thing up and running.'"

‹BILL CROWELL, CIO, MEREDITH CORP.
to fork over an additional 15 percent to 20 percent of what they paid for
the software in annual maintenance fees to cover product updates and tech
support, according to Chuck Phillips, managing director and software
industry analyst at Morgan Stanley Dean Witter. If CIOs want to receive
upgrades, patches and access to tech support‹as inadequate as it can
sometimes be‹they have to pay the yearly maintenance fee. Software companies
earn a significant amount of cash from these fees. So it's in the
manufacturer's best interest, at least financially, to make products that
need maintenance and that have to be continually improved with successive
updates, patches and versions that CIOs pay for up front. In sum, bad
software works for the vendors.

There are, of course, other reasons for all the bugs. IT professionals point
to a whole litany of causes: bloatware, with all its useless bells and
whistles; programmers working in isolation, blissfully ignorant of how
people will ultimately be using their software on a daily basis; reusable
components that may already contain bugs; an absence of agreed upon
professional standards; and developers who take shortcuts to meet deadlines
during development.

But a large part of the story may indeed be the way vendors sell software.
CIOs are finally waking up to this, and a growing number are demanding that
vendors change their business models. A council of IT leaders from a dozen
heavy-hitter enterprises convened in August under the auspices of
Boston-based analyst company AMR Research, intent on pushing for software
industry reform. The group issued a peaceful statement of its desire to
"work with" software companies for improvements in quality, delivery
reliability and versioning. However, with big names like Becton Dickinson,
Boeing, Cabot, General Dynamics and Kraft on the roster, the council has
enough weight to change "work with" to "lean on."

Some other IT users have resorted to more extreme measures‹such as
withholding payments to put pressure on vendors‹but new legislation may soon
make it harder for CIOs to employ such brute financial tactics. The uniform
computer information transactions act (UCITA) makes it harder for customers
to sue vendors and allows vendors to more easily change contract terms. The
UCITA has already been passed in Virginia and Maryland and is under
consideration in seven other states and the District of Columbia.

Fortunately, there are a host of alternative solutions on the horizon, and a
growing number of CIOs are determined to make them a reality. They include
renewable licensing agreements, in which CIOs purchase the right to use
software for two to three years at about 85 percent of the cost of what
they'd pay under a perpetual license. CIOs then have the option to renew the
license at the end of the term if they're happy with the quality of the
product and the support. Subscription licensing agreements are similar to
renewable licenses, except the term is shorter, lasting about a year, and
CIOs rent the software, as opposed to owning it.

Finally, some CIOs are opting to circumnavigate packaged software wherever
possible. They're turning to open-source technologies such as the GNU and
Linux operating systems, the Apache Web server and Sendmail e-mail. "People
are not involved with [the open-source movement] for profit; they're
involved with it because they want to write good product," says Bill
Lessard, coauthor of NetSlaves: True Tales of Working the Web and a former
developer for Prodigy and AOL Time Warner. "If software makers see they are
losing money to people going the open-source route, then they will change.
Until then, it will be business as usual despite appearances."

As much as eight years ago, Patricia Wallington, president of CIO Associates
and former CIO of Xerox, was envisioning a new method of buying software. "I
wanted it to be like a lending library where you could find modules on the
Web, buy the ones you were interested in, cobble them together and create
your own software," she says. "We need to rethink the way we deliver
software because it is so intransigent."

Withholding Payment: The Brute Force Option
The economics underlying the software industry‹its emphasis on quarterly
earnings to impress investors‹leads to the pursuit of short-term profits,
often at the expense of long-term gains. And this tendency has only been
exacerbated by the current market downturn. The revenue of software vendors
is predicated on acquiring new customers. That initial sale provides
software vendors with their biggest profit. So there is a built-in incentive
for vendors to rush a new release of software out the door before it is
completely tested and debugged.

Bill Crowell, CIO of Meredith Corp., the $1.1 billion publisher of Better
Homes and Gardens, believes it is just this profit motivation that has
caused many of the troubles he and other CIOs have had while implementing
Oracle 11i. He had purchased Oracle 10.7SC, a client/server-based financial
system that handled accounts payable, the general ledger and purchasing
functions from the vendor in the spring of 1999, with the assumption that it
would be good for at least four years. But in the fall of 2000, Oracle
released a new Web-based version, Oracle 11i, and told its customers,
including Crowell, that it would be dropping support for all previous
releases. Crowell had no choice but to upgrade. (Under pressure from
customers, Oracle has repeatedly rolled back the end-support date of these
earlier versions. And recently, Oracle officials say they would not
immediately be dropping support for older versions.)

Oracle also promised that 11i would include a feature that would
automatically enter electronic records of all the purchases Meredith
employees had made using their corporate credit cards into the accounts
payable or general ledger system either monthly, weekly or daily depending
on how the company configured it. Crowell says that when he purchased 11i,
the promised functionality was absent. "It wasn't until about a year later
when 11i actually had that capability," he says. It was an inauspicious
beginning. 

As soon as he began the upgrade, Crowell found bugs running rampant in the
software, like ants scuttling over a piece of fruit. Files were corrupted.
Data was lost. Processes didn't work. Screens froze. "It was just a
nightmare," says Crowell. "We were getting literally dozens of developer
patches to this software. Then we were getting patches for patches. The
quality was just atrocious."

One of the biggest bugs bit the interfaces between application components in
the financial system. The system didn't transfer data between accounts
payable and the general ledger, between purchasing and accounts payable and
between purchasing and fixed assets, Crowell says. The failures were bad
enough that had Crowell and his team not been running 11i in a test
environment, Meredith would have had to shut down its financial system. It
would not have been able to do its accounting or pay its bills until the
problems were solved. (Oracle officials declined to comment on either the
bugs in 11i or on Meredith's specific problems with the software.)

"It was clear [Oracle] never tested the interfaces because they flat-out
failed the first time. We felt that what was [supposed to be] their general
release software was effectively beta," says Crowell. "There's no question
that they were under pressure from management to be first with a Web-enabled
version of their software."

The 11i implementation was supposed to go live by April 15, but the bugs
delayed the implementation by just one month, but only because Crowell's
staff worked 24/7 for four months. The CIO estimates that the bugs cost his
company more than $100,000; he had to pay for contractors to help with the
nine-month implementation, and he wasn't able to put staff on other pressing
projects. 

So how did he muddle through this debacle with Oracle? "We didn't pay them,
for one. We owed maintenance of $300,000 to $400,000, and we just didn't pay
it. We said, 'We're holding on to it until you get this thing up and
running,'" Crowell says.

But he thinks it wasn't so much the money that got Oracle to fix the bugs in
11i as it was the brute force he and his project leaders applied in dealing
with the vendor. They called Oracle daily to see if the company was making
headway resolving their problems. They also forced Oracle to give them
contacts in the development group so that they could ask developers directly
for help rather than going through the support team.

Crowell blames economic forces for the problems with Oracle 11i. "They're
trying to move so fast to get the product into the marketplace that they're
not adequately testing and debugging their software," he says. If Oracle had
waited six months before releasing 11i and taken that time to test the
application, he says, the upgrade would have gone off without a hitch.
"Overall we're very pleased with the new application, but if Oracle thinks
they're the Lexus of the software industry, [after] what they've done to
their customers, you feel you've bought a Dodge De Soto," Crowell concludes.

Renewable Subscriptions: Use Now, Pay Later
Crowell believes that the new renewable and subscription arrangements that
are becoming more prevalent in the software world would have ameliorated the
problems he ran into while deploying 11i. Under a subscription model, in
which he would have paid

"If CIOs could say, 'You'll get 10 percent now and 10 percent after each
quarter if the software works, it would give vendors a financial incentive
to make sure the product works."

‹GREG SEYK, CIO, VISIONQUEST
less up front, Crowell would have had more leverage. Also, it would have
given Oracle a greater financial incentive to please Crowell. In fact,
Crowell plans to start buying software from Microsoft on a subscription
basis in two years, once he finishes receiving all the upgrades he paid for
two years ago. To him, the renewable model makes sense. It's the way his own
publishing industry works. "It's a subscription. We know when the revenue is
coming in. We can plan our business around it," he says. "And we deliver a
quality product every month. The [vendors] need to think about delivering
quality every month and a business model that allows them to do that."

Microsoft, in fact, announced that it would begin offering a brand-new
subscription license this month for its operating systems and software,
including Microsoft Windows Professional and Microsoft Office Professional.
The Enterprise Agreement Subscription, as the new license is called, is a
major departure from the perpetual model. CIOs will now lease the software
under subscription licenses. While CIOs see the potential benefits of the
subscription model, many are uncomfortable with the specific terms Microsoft
is offering. For instance, Microsoft is requiring that customers pay a hefty
annual fee even before new upgrades are released.

Microsoft has also introduced a new and more controversial twist to its
perpetual Open and Select licensing agreements. These programs pressure CIOs
to upgrade to new versions such as Windows XP by Feb. 1, 2002, at a
discounted rate. If they don't, they will have to pay twice as much to
upgrade after that date. (For more information on this controversy, go to
"Looks Can Be Deceiving").

Mark Grove, CIO of AmericasDoctor, a pharmaceutical services company based
in Chicago, says software vendors that require perpetual licenses with
constant upgrades are not serving their customers. "Any vendor who's doing
that is trying to force the customer to follow the vendor's business model,"
Grove says. "When they tell a customer that he has to upgrade at a certain
time, they're forgetting that their customers have their own business cycles
and busy seasons that they have to work within."

After his experience with Lawson, VisionQuest's Seyk is also considering
buying software from vendors on a renewable or subscription basis when it is
offered. He likes the idea of not paying for the software entirely up front.
"Once you give [vendors] the cash and the software doesn't perform, you have
no leverage," he says. "If [CIOs] could say, Sure, you'll get 10 percent now
and 10 percent after each quarter, and a year from now you'll get it all if
[the software] works," that would be a way to hold the vendor accountable.
"It gives [the vendor] a financial incentive to make sure the product
works," he adds. 

Open Source, Open Sesame
Some CIOs believe open source could liberate businesses from their
dependence on the dysfunctional software-making ma-chine. Linux now boasts
important implementations at companies including Shell Oil, hotel franchiser
Cendant, networking giant Cisco and retailer Burlington Coat Factory, while
the Apache program is by many accounts the most widely used Web server
software today. The prevalence of these open-source packages will only
continue to grow, with big IT vendors such as IBM providing hardware with
Linux and Apache bundled in.

Raymond Dury, co-CIO of Ameritrade Holding, says open-source
"Those folks involved in the open-source movement are very knowledgeable at
what they do, and they're producing really great code."

‹RAYMOND DURY, CO-CIO OF AMERITRADE
software, because of its community-style development method, has flexibility
that commercial software lacks. The packaged software market doesn't offer
the solutions he needs to support his company's thousands of daily online
transactions. To enable customers to make trades faster, hold more trades in
their portfolio and get real-time quotes, for example, Ameritrade's website
needs enormously robust software code‹code that can support tens of
thousands of simultaneous connections with sub-second responses.

Instead of buying software and additional hardware to extend the speed and
computing power of his website, Dury is looking to configure his own
software using open-source code to support all of those transactions.

He also wants to replace three separate software components that communicate
back and forth when a customer wants to execute a trade with one piece of
proprietary software built using open-source code to consolidate these
functions. "We have separate pieces of software communicating with each
other, trying to do one process," Dury says. "Instead we could use a piece
of open source to consolidate these separate functions under one piece of
software." 

Dury has long been a fan of open source. He used it to develop a secure
e-mail application while serving as the vice president of operations at
Netdox, a Deloitte and Touche venture in 1997. "Those folks [involved in the
open-source movement] are very knowledgeable, very good at what they do, and
they're producing really great code," says Dury.

Open source would give CIOs the flexibility to build exactly what they need,
he says. However, many are skeptical about using it because the technology
is immature, and it's hard to find programmers who know how to write the
code and maintain it. When you buy software from a vendor, you can always
turn to its help desk, however incompetent. With open source, you're on your
own. 

Dury acknowledges that open source requires seasoned in-house IT staffs‹who
know how to build and integrate their own systems‹and that not all companies
can afford to hire this kind of talent. But he likes open source for the
same reason that Crowell and Seyk like the renewable options to perpetual
licensing agreements. "If you increase competition [in the market] because
there's quality in open-source software and because it's low cost, some
folks will move toward it," Dury says. "As a result, vendors are probably
going to have to increase their quality and reduce their costs."

What do you think will improve the quality of packaged software and the
practices of software vendors? E-mail Senior Writer Meridith Levinson at
mlevinson@cio.com. 

PHOTO CREDITS: BILL CROWELL BY DOUG 

------------------------ Yahoo! Groups Sponsor ---------------------~-->
Pinpoint the right security solution for your company- Learn how to add 128- bit encryption and to authenticate your web site with VeriSign's FREE guide!
http://us.click.yahoo.com/yQix2C/33_CAA/yigFAA/kgFolB/TM
---------------------------------------------------------------------~->

------------------
http://all.net/ 

Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/ 



This archive was generated by hypermail 2.1.2 : 2001-12-31 20:59:55 PST