Billions of dollars from pension funds and other nontraditional players have been moving into the reinsurance business in recent years, according to a report released on Wednesday by the Treasury Department.