Spokesperson says PA president had family commitments due to Muslim holiday and informed visitors 5 days ahead of meeting, willing to meet GOP members next time