The public key is just that, public, by definition.

If you go to any website on the internet you can download any servers public key from its SSL certificate.

The security with SSL pinning comes from verifying that the certificate being presented to you has a public key you trust. There's more to it than just that as you additionally trust the certificate because you can build up a chain of trusted certificates to a root certificate installed on the device, so really its the combination of these two parts. The trust with the chain of certificates comes as the parents private key is used to sign its children.