Scott took the kids to explore Blarney Castle while the rest stayed in Blarney town