Enable authentication and authorization with the Spark History Server charm

Charmed Apache Spark includes the Spark History Server charm, which lets users monitor application workflows and logs. By default, Spark History Server does not provide authentication or authorization, both of which are essential in production environments. To address this limitation, you can integrate the Spark History Server charm with the Canonical Identity Platform bundle.

This guide assumes you already deployed Charmed Apache Spark as described in the Charmed Apache Spark deployment guide, including a Spark History Server charm configured with an object storage backend.

In this guide, we will use the variable $SPARK_MODEL as a placeholder for the Juju model where the Spark History Server charm is deployed. If you want to copy the commands in the following sections and use it, make sure you set the value of $SPARK_MODEL as the actual model where the Spark History Server charm is deployed. For example:

export $SPARK_MODEL="spark"

Enable Authentication

Authentication for Spark History Server charm is provided by the Canonical Identity Bundle.

Deploy the Canonical Identity bundle

Deploy the Canonical Identity bundle by following this tutorial.

The deployment will create two Juju models where Identity Platform charmed applications and their charm dependencies are deployed, configured and integrated.

The iam model contains all the crucial identity applications:

And the core model contains all of their shared dependencies:

Configure the identity provider

You must also configure the identity provider you want to use. You can either use the built-in identity provider that is enabled by default in Charmed Kratos, or use an external identity provider.

This guide uses the local identity provider via Charmed Kratos. Follow this guide if you want to use an external identity provider.

Create your personal admin account:

juju run -m iam kratos/0 create-admin-account email=<your-email> username=<username>

After the account is created, use the provided password reset link to set your password and complete the setup. You can now use this account to log in to the Spark History Server.

For supported identity providers and additional details, see the How to manage external identity providers guide.

Deploy and integrate Charmed OAuth2 Proxy

The connection between Spark History Server and the Identity Platform is handled by the Charmed OAuth2 Proxy charm. OAuth2 Proxy protects endpoints exposed through an ingress (for example, Traefik).

Deploy OAuth2 Proxy charm and integrate it with Spark History Server:

juju switch $SPARK_MODEL
juju deploy oauth2-proxy-k8s --channel latest/stable --trust
juju integrate oauth2-proxy-k8s spark-history-server-k8s:oauth2-proxy

Now, consume the oauth-offer offered from the iam model, and integrate OAuth2 Proxy with it:

juju consume iam.oauth-offer
juju integrate oauth2-proxy-k8s:oauth oauth-offer

Finally, consume the send-ca-cert offered from the core model, and integrate it with Oauth2 Proxy over the receive-ca-cert relation endpoint:

juju consume core.send-ca-cert
juju integrate oauth2-proxy-k8s:receive-ca-cert send-ca-cert

Configure ingress

An ingress needs to be deployed and configured such that it forwards authentication requests to Charmed OAuth2 Proxy charm, and only the requests that are completely authenticated are passed to the Spark History Server charm.

Spark History Server charm currently supports two Ingress provider charms (namely the Traefik ingress and the Istio ingress), depending upon whether it is added to Istio service mesh or not.

Traefik Ingress (non meshed setup)

For a non meshed setup, the Traefik ingress that comes already bundled in the Identity Platform bundle can be used. To use it, first configure the existing Traefik ingress to enable the forward-auth feature, and expose the forward-auth offer.

juju switch core
juju config traefik-public enable_experimental_forward_auth=True
juju offer traefik-public:experimental-forward-auth traefik-forward-auth

Also offer the endpoint ingress from the core module:

juju offer traefik-public:ingress traefik-ingress

Now, switch back to the model $SPARK_MODEL and integrate the ingress with the Spark History Server and the OAuth2 Proxy charms.

juju switch $SPARK_MODEL
juju consume core.traefik-ingress
juju integrate spark-history-server-k8s:ingress traefik-ingress
juju integrate oauth2-proxy-k8s:ingress traefik-ingress

Integrate the OAuth2 Proxy charm with the traefik-forward-auth offer over the forward-auth relation endpoint:

juju consume core.traefik-forward-auth
juju integrate oauth2-proxy-k8s:forward-auth traefik-forward-auth

After integration completes, get the endpoint by running:

juju run -m core traefik-public/leader show-proxied-endpoints

You should see an output similar to the following:

proxied-endpoints: '{"traefik-public": {"url": "https://10.99.99.0"}, "remote-xxxxxxxxxxxx": "http://10.99.99.0/testmodel-spark-history-server-k8s/"}'

The URL endpoint for the Spark History Server is the one ending with spark-history-server-k8s (or the name of the Spark History Server app).

Istio Ingress (Istio service mesh setup)

Note

Supported only for charm revisions greater than 143 (arm64) and 144 (amd64) on track 3/.

If the Spark History Server is running behind Istio service mesh, Traefik is not supported as Ingress and therefore, Istio ingress should be used. Istio ingress requires the Istio control plane to be installed in the cluster.

Deploy istio-ingress-k8s charm:

juju switch $SPARK_MODEL
juju deploy istio-ingress-k8s --channel 2/stable --trust

Note

The istio-ingress-k8s requires istio-k8s properly deployed in order to work. It is assumed that if you have a Istio service mesh setup, you already have a working istio-k8s deployment. If not, follow this guide to deploy istio-k8s and enable service mesh for Spark History Server.

Integrate istio-k8s and istio-ingress-k8s over the istio-ingress-config relation endpoint:

juju integrate istio-k8s:istio-ingress-config istio-ingress-k8s:ingress-config

Integrate istio-ingress-k8s with the certificates offer, such that HTTPS is enabled:

juju consume core.certificates
juju integrate istio-ingress-k8s:certificates certificates

Integrate the Istio ingress charm with OAuth2Proxy and Spark History Server charms:

juju integrate istio-ingress-k8s:ingress-unauthenticated oauth2-proxy-k8s:ingress
juju integrate istio-ingress-k8s:forward-auth oauth2-proxy-k8s:forward-auth
juju integrate istio-ingress-k8s:ingress spark-history-server-k8s:ingress

After the integration completes and the charms are in active and idle state, get the Juju status of the istio-ingress-k8s app.

juju status istio-ingress-k8s

You will see a message similar to the following under the Message column in the Juju status:

Serving at <ip-address>

This is the address for the Ingress gateway. Once you find the ingress gateway address, the URL endpoint for the Spark History Server should be at the following path:

https://<ingress-gateway-address>/<juju-model-name>-spark-history-server-k8s

Access Spark History Server UI

Once you find the URL endpoint for the Spark History Server using the ingress of your choice, open the URL in the browser. When you open the URL, you are redirected to your configured identity provider for authentication. After successful login, you can access the Spark History Server UI.

Authorization Management

By default, all authenticated users can access Spark History Server. To restrict access, configure an allow-list of authorized users. Provide email addresses as a comma-separated list:

juju config spark-history-server-k8s authorized-users="user1@canonical.com,user3@canonical.com"

Oathkeeper integration (deprecated)

Previously, authentication in this guide used the oathkeeper charm. oathkeeper is now deprecated in favor of Charmed OAuth2 Proxy.

If your deployment still uses oathkeeper, migrate by removing integrations with oathkeeper, updating Identity Platform components as described above, and then integrating oauth2-proxy-k8s.