Investors go to specialist lenders as banks pull back