The release of minutes of the April 30-May 1 meeting reversed a surge earlier in the day on hopes that the Fed was a long way from stopping its stimulus efforts.
The Dow Jones industrial average was down 33 points, or 0.2 percent, to 15,354 as of 2:50 p.m. Eastern Daylight Time. Earlier in the day, it had risen 154 points after testimony to Congress by Fed chairman Ben Bernanke raised investors' hopes.
"If you had any doubts about the influence of the Fed (on stocks), you only have to look at the roller coaster that followed Bernanke's testimony this morning and the release to Fed minutes this afternoon," said David Kelly, chief global strategist at J.P. Morgan Funds.
The Standard and Poor's 500 fell 9 points to 1,659, a decline of 0.6 percent.
Minutes of the Fed's meeting showed that "a number" of members expressed a willingness to scale back the $85 billion a month in Treasury and mortgage bonds the Fed has been purchasing, perhaps as soon as June, if the economy accelerates. The Fed next meets June 18-19.
Earlier in the day, Bernanke had told lawmakers it was too soon for the central bank to pull back on its stimulus programs. That sent stocks broadly higher. Investors were also encouraged by news that sales of previously-occupied U.S. homes rose last month to the highest level in three and a half years.
"It's up, up and away," said Stephen Carl, head of stock trading at the Williams Capital Group, shortly after Bernanke stopped speaking.
In addition to buying bonds, the Fed has been keeping short-term interest rates near zero to encourage people and businesses to borrow and spend more.
The National Association of Realtors said Wednesday that sales of previously occupied homes rose to a seasonally adjusted annual rate of 4.97 million in April, helped by a jump in the number of homes available for sale.
Among stocks making big moves:
—Bristol-Myers Squibb jumped 5 percent, or $2.24, to $46.30 after a Citigroup analyst raised his rating on the drugmaker. The analyst said the company could be a big winner with a group of cancer treatments under development.
—Saks rose $2 to $15.67, or 15 percent, after The New York Post reported the luxury retailer had hired Goldman Sachs to explore options for the company, including a possible sale. A spokesman for Saks declined to comment.
—Target fell $3.57, or 5 percent, to $67.69 after announcing a 26 percent drop in first-quarter profits. The company also said full-year earnings may come in lower than previously expected.
On Tuesday, stocks rose after James Bullard, president of the St. Louis branch of the Federal Reserve, told an audience in Germany that the central bank should continue buying bonds.
The Russell 2000 index of small-company stocks fell 13 points to 985, a loss of 1 percent.
The Nasdaq composite was down 32 points at 3,469, or 0.9 percent.
The price of gold fell $10.20 to $1,367.40 an ounce, a drop of 0.7 percent. Crude oil fell $2 to $94.16 a barrel on the New York Mercantile Exchange.