View Larger

Lowell industrialist Frederick Ayer Sr. and his descendants built a $600 million fortune over more than a century. But a family confidant allegedly siphoned about $58 million away in a few years, exploiting his position with the family office that manages the estate, according to a lawsuit by the firm.

That case, being played out in the Business Litigation Session of Suffolk Superior Court in Boston, offers a rare glimpse at the inner workings of the clubby and highly guarded world of family offices, the corporate entities often tasked with managing the day-to-day operations of large family trusts and other wealthy estates. The saga also highlights a little discussed but unavoidable fact in the arena of wealth management: Many family offices are vulnerable to fraud, especially by trusted insiders.

“It’s a fear that every wealth owner has,” said John Benevides, president of Family Office Exchange LLC, a Chicago company that advises more than one-tenth of the estimated 3,500 family offices in North America.

Adds Carrie Seligman, director of U.S. Trust’s estate and financial planning division in Boston: “If a trustee is bad, fraud isn’t difficult. It’s like giving away the keys to the safe.”

In the case of the Ayers, court documents and interviews indicate the trusts may have been especially vulnerable because of the complexity of the estate, relaxed oversight and what the family claims are auditing failures.

Tenens Corp., the family company charged with managing the estate for more than 100 relatives, operated in relative obscurity for years under the name Essex Street Advisors from a woodsy, non-descript group of offices in Beverly.

But the tranquility was shattered in March 2006 when Essex Street CEO Caleb Loring III terminated a longtime employee, John F. Doorly, after learning about a suspicious transaction, according to court filings.

Within days, Essex filed suit against Doorly, accusing him of stealing millions of dollars from the family fortune — money he allegedly spent on everything from a Gulfstream jet to gifts for his mistress and family, including a luxury condo in Boston’s Back Bay for his son’s girlfriend.

Doorly, 57, denied stealing any money, insisting many of the withdrawals were used for loans and investments that are still earning money for the Ayer estate. In fact, he argues that the jet and other properties were long on the family’s books, but were simply unnoticed due to lax oversight.

Doorly, a graduate of St. Mary’s High School in Lynn, joined Essex Street in 1973, when he was a self-described computer operator. He never earned a college degree, but he eventually became the company’s chief operating officer, managing most of the company’s daily operations, including the payroll and some of the family’s investments.

But over the last 14 months, Essex Street said in the lawsuit, its forensic accountants uncovered evidence that Doorly used a variety of schemes to steal more than $58 million, far more than Essex initially thought was missing.

Even more troubling, Essex Street alleged the embezzlement began at least a decade ago without detection.

According to court papers, it appears Ayer family members didn’t identify the missing money earlier for several reasons. Among them:

• The complexity of the trust. According to the lawsuit, Essex Street maintained a network of 300 trusts for more than 100 family members, making money difficult to track.

Rather than operate as a single trust, the Ayer estate was set up as a so-called “separate shares” trust, whereby new trusts were added as the family tree grew. And each of the trusts requires trustees. But trust experts say separate-shares trusts can be complicated as the number of beneficiaries grows, as trustees often run into roadblocks when making decisions that require input from all family members.

“Some beneficiaries are very passive,” said Bob Holdway, a vice president with Fiduciary Trust in Boston, speaking generally. “As a good trustee, you try and reach out to them. But there are always some who are used to running on auto pilot.”

• Inadequate oversight. Doorly testified that Loring, the CEO, and other trustees were rarely in the office, leaving Doorly to manage the operation largely on his own.

“I would be the first one in, last one out, seven days a week averaging 70 hours doing my work as well as work that should have been performed by Caleb Loring III,” Doorly said in his sworn affidavit.

A Loring associate, speaking on the condition he not be identified, said the characterization is unfair.

• Auditing failures. Although Essex hired outside auditors, it appears they didn’t catch the missing money. Essex accused one auditor, Vitale Caturano & Co., of following Doorly’s instructions to ignore many of the accounts, leaving Essex Street vulnerable. Vitale said it did what it was hired to do.

Essex has sued Vitale and others in the case to try to recoup the missing money. But it has saved most of its ire for Doorly, calling him in a prepared statement a trusted employee who “abused that trust by looting the family’s cash accounts to enrich himself.” The family also vowed to pursue any parties who benefited from or aided the alleged theft.

Benevides says there are a number of safeguards families can put in place to protect themselves, including annual audits, strong internal financial controls and the screening of new hires. His firm’s research indicates that nearly three-quarters of its members had audits in the last two years, though audits can vary widely in scope.

But Benevides warns: “As any auditor will tell you, if someone has the intent to defraud, it is very difficult to detect.”

Essex Street has recovered around $9 million in cash and real estate from Doorly and Doorly-affiliated entities, according to court filings. Suffolk Superior Court Judge Allan van Gestel recently ruled that Doorly defied a court order to disclose and freeze his total income and assets. Doorly has since been forced to roll hundreds of thousands of dollars in cash into an escrow account.

It’s unclear, however, how much more Essex Street will be able to recoup. As of January, Doorly said he had an annuity worth $1.4 million, a time share at the Ritz Carlton in Jupiter, Fla., and two Cadillacs and a Jaguar. He also has claimed that Essex Street’s accounts hold around $2.5 million of his money.

Meanwhile, his personal life appears to be unraveling. Earlier this year, Doorly’s wife of 34 years, Maryjane Doorly, divorced him after learning through court filings that Doorly had a mistress. And as of a few months ago, John Doorly, who now lives in Peabody, was unemployed.

Meanwhile, the office of U.S. Attorney Michael Sullivan has issued a number of subpoeanas as it investigates the matter, a person directly involved in the case said.