Shapiro, who has lost at least $545 million to Madoff, is one of numerous large investors who are under investigation by US authorities.

And now, as a client who has known Madoff for five decades, Shapiro has to worry if the court-appointed trustee recovering funds for victims will try to seize any profits he or his charitable foundation received from him over the years, as the trustee has sought to do with other large clients.

“There’s a significant risk for people who had substantial accounts for many, many years, who were looking at the supposed performance regardless of what the market was doing and seeing gains year after year,’’ said Boston attorney Harry Miller, who represents a number of Madoff victims. “The trustee is going to take the position that they should have known, and could hold them responsible for that.’’

And if regulators determine he received unusually large returns over his many years with Madoff, the consequences could be tougher still.

Irving Picard, the Madoff bankruptcy trustee, is pursuing a number of large investors for “clawbacks,’’ or demands they return profits from Madoff because the money, in effect, belonged to other investors.

He is limited to the past six years of gains; the Securities and Exchange Commission has no time limit on how far back it can go to recover what it calls “ill-gotten gains.’’

Picard and the Shapiros have not commented on whether the family has received a clawback demand.

In some ways, life continues as usual for two of the men closest to the Wall Street swindler. They have returned from Palm Beach to the Boston area for the summer, as they typically do. There was the recent wedding of the Jaffes’ son Steven, and they recently attended the bar mitzvah of the child of some friends.

But in other ways, life is changing in ways big and small. Jaffe will not be a regular at the Pine Brook Country Club in Weston this season; he’s taken a year off from the golf club “for financial reasons,’’ his spokesman said.

Meanwhile, Shapiro, 96, has been pained to see a large part of his personal fortune wiped out, along with half of the charitable foundation that has donated in his family’s name to hospitals, art museums, and schools in Greater Boston and Palm Beach. He has said he was as shocked as anyone by the scandal.

The Shapiros have donated $196 million to the Carl and Ruth Shapiro Family Foundation over the past decade, according to the foundation’s tax records filed through 2007. More than half of that, $111.5 million, was donated in 2007.

The family uses another entity to sometimes direct contributions to the foundation: Wellesley Capital Management Inc., which accounted for $49 million of the donations during the 10-year period. The firm was established in 1975 to handle tax and accounting services for the family fortune and keeps the books for the foundation, according to tax records and state filings. It is also listed as a client on Madoff customer lists.

Through a spokesman, the Shapiros declined to say if the money they gave the foundation came from Madoff or other sources.

Last week, a Shapiro family confidant who asked to remain anonymous confirmed that both the US attorney in New York and the bankruptcy trustee are examining Shapiro’s investments with Madoff.

Wellesley Capital did not have oversight of Shapiro investments, said the family spokesman, Elliot Sloane, but is a “bookkeeping and accounting office serving the needs of the Shapiro family investments.’’ It has a small staff, and its officers are Shapiro’s three daughters: Linda Waintrup, listed as president; Rhonda Zinner, and Ellen Jaffe, Robert’s wife.

On Madoff’s customer list – flawed and error-ridden though it is – no entity is mentioned more times than Wellesley Capital Management. The firm, or the address and suite number of its office, appears 129 times, with clients that include Shapiro family trusts and two of the Jaffes’ sons.

Other investors with longtime relationships with Madoff also appear multiple times on the list: Jeffrey and Barbara Picower and their foundation, which Picard has alleged took billions more in money out of Madoff accounts than they put in, show up 10 times, and New York money manager Ezra Merkin, who funneled $2.4 billion in client money to Madoff, shows up eight times. Stanley Chais, the Beverly Hills money manager charged with fraud by the Securities and Exchange Commission, is listed 68 times.

One possible explanation for the large number of Shapiro family accounts on the client list is that Carl Shapiro’s relationship with Madoff dates back to the 1960s.

People who know Shapiro said he thought of Madoff as a son. In the days before the collapse of his scheme in December, Madoff asked Shapiro for $250 million, which his friend gave him. Shapiro learned of Madoff’s confession on the television news, say people who know him.

Known for an exacting attention to detail, Shapiro has kept a firm hand in the workings of his charity and his finances, according to people who know the family. He takes a personal interest in many of the nonprofits the foundation funds. He set up Wellesley Capital rather than hire an outside firm to manage his affairs. And for years he used a New York accounting firm run by a friend of Madoff’s to prepare the foundation’s taxes, which often were filed late.

The question many are asking is this: How could Shapiro or other Madoff investors have failed to see that something was amiss when they received returns that beat the market so consistently?

Miller said part of the answer lies in human nature. “If things were really good, you might look the other way and not look into what was going on,’’ he said.

It appears the Shapiros are cutting their Madoff ties one by one. The foundation has fired the accounting firm Konigsberg Wolf & Co., the family’s spokesman said.

Neither Shapiro nor Jaffe attended Madoff’s court appearances, nor did they submit character references to the judge who sentenced Madoff.

Indeed, not a single person did so on his behalf.

Beth Healy can be reached at bhealy@globe.com. Steven Syre can be reached at syre@globe.com.