Will the NBA make Donald Sterling the first owner to give up team?

Adam Silver has been NBA commissioner for only a few months. A few relatively controversy-free months.

Now -- with the longtime owner of a top team alleged to have made racist remarks in a recorded conversation posted on a celebrity-focused website -- Silver is stepping down a new road.

Only one owner has been suspended in 68 years. That was Minnesota's Glen Taylor in 2000, when the Timberwolves made a secret deal with a star player to circumvent salary cap rules.

The league has never forced an owner to give up his team.

Should Los Angeles Clippers owner Donald Sterling face either of those punishments for the recording that TMZ.com says is him talking to his girlfriend V. Stiviano?

First, the NBA says, it will investigate the matter to determine whether the recorded conversation is authentic, whether it is actually Sterling and whether it has been edited or manipulated.

"All members of the NBA family should be afforded due process and a fair opportunity to present their side of any controversy, which is why I'm not yet prepared to discuss any potential sanctions against Donald Sterling," Silver told reporters Saturday.

Silver said the league will move "extraordinarily quickly" in its investigation and he hoped to have an initial report in "the next few days."

He indicated the NBA had been in contact with Sterling already, saying the league's longest-tenured owner had agreed not to travel to Oakland, California, on Sunday where the Clippers are playing the Golden State Warriors in the first round of the NBA playoffs.

There is no clear answer as to what the discipline should be, if the accusations against Sterling -- who has yet to comment publicly -- are true.

"There are broad powers in place under the NBA's constitution and bylaws that include a range of sanctions and all of those will be considered depending on the findings of our investigation," Silver said Saturday.

According to the website TMZ, Sterling made the alleged discriminatory remarks during an argument he had with girlfriend V. Stiviano on April 9. TMZ posted a 10-minute recording of the argument online Friday. The law firm representing Stiviano said the recording is a portion of an hourlong conversation between the couple.

If authentic, the remarks seem to reflect Sterling's embarrassment and frustration with Stiviano over her associating with African-Americans at Clippers games and for posting such pictures on her Instagram account.

"This is a failure of leadership on so many levels," Jason Johnson, an HLN contributor and a political science professor at Hiram College, said. "This is not the first time we have had an owner get in trouble, and they should have had rules in place to deal with these kinds of situations."

The most likely penalties, said CNN Sports anchor Andy Scholes, are a fine and a suspension, possibly banning the 80-year-old Sterling from team operations for as long as a year.

TNT analyst Charles Barkley said there's no question Silver should suspend Sterling soon.

"He's got to suspend him right now," Barkley said Saturday night. "First of all, they have to prove it is his voice on the tape, but this is the first big test of Adam Silver. You can't have this guy making statements like that. He has to suspend him and fine him immediately."

Sterling has never been sanctioned by the league since he bought the team in 1981.

Other owners have been fined, most notably Dallas Mavericks owner Mark Cuban, who has paid out nearly $2 million in 19 fines, according to Bleacher Report. One fine reportedly was $500,000, about the same amount Miami Heat owner Micky Arison supposedly had to pay in 2011. But those were for remarks about officiating (Cuban) and the lockout (Arison).

And a similar record-level fine for the Clippers owner might not seem like much to the public, since Sterling is worth billions.

The league probably cannot force Sterling to give up the team, which according to Forbes magazine is worth $575 million, Scholes said.

According to Scholes, the NBA constitution is confidential, and reportedly the league would only step in to remove an owner if the team were in financial trouble.

The value of the Clippers, who were a terrible team for decades, has been on the rise with their recent success on the court. Led by all-stars Blake Griffin and Chris Paul, who is also the head of the players union, the team has set win records in the past two seasons. The other players will look to Paul for guidance, but a players walkout is unlikely.

"They are looking at Chris Paul and they are trying to see what can they actually do to show their disdain and dislike (for Sterling)," Johnson said.

The NBA will face an image problem if Sterling turns out to be the voice on the recording, in which a man tells a woman that she shouldn't be seen at Clippers games with Earvin "Magic" Johnson because he is African-American.

The question would become: Could the Clippers sign any free agents in the offseason if Sterling remains the owner?

LeBron James, who could potentially be one of those free agents, indicated he wouldn't consider playing for Sterling's Clippers.

"There's no room for Donald Sterling in the NBA," he said Saturday before the Miami Heat played the Charlotte Bobcats. "There is no room for him."

James told ESPN he was certain Silver would take care of the matter.

Jason Johnson said Silver needs to be forceful.

"He can say, 'Look, you have damaged the brand. You have damaged the league, and consequently, I think we are going to have to look at your right to own this team,' " Johnson said.

Sterling has faced questions tied to racial issues in the past. In 2009 he settled -- while admitting no liability -- three lawsuits tied to discrimination claims at apartments he owned in Los Angeles County, the Los Angeles Times reported. He also settled a 2003 lawsuit that claimed he wouldn't rent to Latinos, the Times reported.

Sterling was sued in 2009 by former Clippers general manager Elgin Baylor, who said he was a victim of age and racial discrimination. Baylor dropped the race claim and a jury ruled in Sterling's favor.

Silver said Saturday that Sterling's past "hasn't so far" played into the investigation. "We're very focused on the audio tape ... and the facts around it."

The NBA playoffs will continue through mid-June, when the league will crown a champion.