Expose the Spark History Server UI

Without Ingress (MicroK8s only)

The Spark History server exposes a UI accessible at http://<spark-history-server-ip>:18080.

If you are running MicroK8s, you can directly expose it to the local network by enabling DNS

microk8s enable dns

and retrieve the Spark History server POD IP using

IP=$(kubectl get pod spark-history-server-k8s-0 -n spark --template '{{.status.podIP}}')

With Traefik Ingress

The Spark History server can be exposed outside a K8s cluster by means of an ingress. This is the recommended way in production for any K8s distribution. Exposing Kubernetes services through an ingress generally requires the cloud provider/infrastrucutre to have an external load balancer integrated with the Kubernetes cluster. Most cloud providers (such as AWS, Google and Azure) provide this integration out-of-the-box. If you are running on MicroK8s, make sure that you have enabled metallb, as shown in the “How-To Setup K8s” user guide.

Spark History server can be exposed outside of the K8s cluster using traefik-k8s charm. If COS is enabled, you can use the ingress already provided as part of the COS bundle. Otherwise, you can deploy one using

juju deploy traefik-k8s --channel latest/candidate --trust

Then, integrate with the Spark History server charm

juju integrate traefik-k8s spark-history-server-k8s

After the charms settle down into idle/active states, fetch the URL of the Spark History server with

juju run traefik-k8s/0 show-proxied-endpoints

This should print a JSON with all the ingress endpoints exposed by the traefik-k8s charm. To also exposed the UI outside the local cloud network via a public domain or to enable TLS encryption, please refer to this guide for Let’s Encrypt certificates about integration of traefik-k8s with Route53 and Let’s Encrypt (note that this is currently only supported on AWS EKS only).

With Istio Ingress (meshed setup)

Note

Supported only for charm revisions greater than 143 (arm64) and 144 (amd64) on track 3/.

The Spark History Server needs to be accessed using the Istio Ingress if it is running behind Istio service mesh. Follow this guide for more information on how to access the Spark History Server using Istio Ingress.