#openobserve

ZincObserve Installation Issue - Email and Env variables

TLDR Jens faced issues after installing ZincObserve via Helm chart due to ZO_ROOT_USER_EMAIL and ZO_ROOT_USER_PASSWORD not set. Prabhat suggested trying the latest dev container and changing to a simpler password. The issue was resolved after changing the email which didn't match the regex validation.

Powered by Struct AI
๐Ÿ‘ 5
๐ŸŽ‰ 2
๐Ÿคฃ 1
123
1w
Solved
Join the chat
May 25, 2023 (1 week ago)
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:07 PM
Hi, I installed zincobserve via helm chart to my cluster and all components claim that ZO_ROOT_USER_EMAIL and ZO_ROOT_USER_PASSWORD are not set - both values are configured in the generated secret and the secret is mounted as env source to the pod. Any hint what could be wrong?
Image 1 for Hi, I installed zincobserve via helm chart to my cluster and all components claim that <code>ZO_ROOT_USER_EMAIL</code> and <code>ZO_ROOT_USER_PASSWORD</code> are not set - both values are configured in the generated secret and the secret is mounted as env source  to the pod. Any hint what could be wrong?Image 2 for Hi, I installed zincobserve via helm chart to my cluster and all components claim that <code>ZO_ROOT_USER_EMAIL</code> and <code>ZO_ROOT_USER_PASSWORD</code> are not set - both values are configured in the generated secret and the secret is mounted as env source  to the pod. Any hint what could be wrong?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:10 PM
Somehow the secret did not get mounted on to ZincObserve pods
02:10
Prabhat
02:10 PM
Can you check what env variables are set for any ZO pod
02:11
Prabhat
02:11 PM
actually this is from the pod. right?
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:12 PM
Yes
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:12 PM
Also is this screenshot from Lens?
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:12 PM
YEs it is
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:12 PM
This is my screenshot
Image 1 for This is my screenshot
02:13
Prabhat
02:13 PM
probably could be due to different lens versions
02:13
Prabhat
02:13 PM
so you must be using release version
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:13 PM
Image: public.ecr.aws/zinclabs/zincobserve:v0.4.3
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:13 PM
Can you try with the latest dev container
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:14 PM
Iโ€™ll try
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:14 PM
also if you can share your values.yaml file
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:16 PM
USER-SUPPLIED VALUES:
auth:
  ZO_ROOT_USER_EMAIL: [email protected]*****.com
  ZO_ROOT_USER_PASSWORD: YP...r
  ZO_S3_ACCESS_KEY: AKI...
  ZO_S3_BUCKET_NAME: zinc-20230525135027032600000002
  ZO_S3_SECRET_KEY: O...
certIssuer:
  enabled: true
ingester:
  persistence:
    accessModes:
    - ReadWriteOnce
    enabled: true
    size: 10Gi
    storageClass: gp3-encrypted
    volumePermissions:
      enabled: false
ingress:
  className: nginx-internal
  enabled: true
  hosts:
  - host: zinc.k8s.sumcumo.dev
    paths:
    - path: /
      pathType: ImplementationSpecific
  tls:
  - hosts:
    - zinc.k8s.sumcumo.dev
    secretName: k8s-sumcumo-dev-wildcard-tls
minio:
  enabled: false
replicaCount:
  alertmanager: 1
  compactor: 1
  ingester: 1
  querier: 1
  router: 1
02:16
Jens
02:16 PM
There is not too much configuration yet
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:16 PM
going through it
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:18 PM
The password contains a few characters that might confuse a shell. could this be an issue (I havenโ€™t inspected the container in detail yet)
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:18 PM
Don't know. Give a min.
02:20
Prabhat
02:20 PM
so there isn't really anything that could cause an issue
02:20
Prabhat
02:20 PM
maybe let's try a simpler password just to confirm
02:21
Prabhat
02:21 PM
actually you have etcd removed
02:21
Prabhat
02:21 PM
You plan to use ZincObserve in cluster mode. right?
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:22 PM
That is just the user supplied part, etcd is deployed with the default values
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:22 PM
ohk
02:23
Prabhat
02:23 PM
then there is nothing really that can break.
02:23
Prabhat
02:23 PM
Let me DM you.
02:25
Prabhat
02:25 PM
Am glad that changing to a simpler password and going to dev image worked.
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:26 PM
Too early to say. the chart is still updating
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:26 PM
๐Ÿ™‚
02:27
Prabhat
02:27 PM
We can wait ๐Ÿ˜…
02:28
Prabhat
02:28 PM
BTW, if you are on AWS , I would recommend you to use IRSA
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:36 PM
ATM we use an external OIDC-IdentityProvider and it seems that IRSA requires the cluster to use the AWS-provided OIDC-Issuer IIRC
02:37
Jens
02:37 PM
Unfortunately the image still complains about the missing env
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:37 PM
Hmm
02:38
Prabhat
02:38 PM
and crashes. right?
02:38
Prabhat
02:38 PM
is the secret created?
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:38 PM
Yes
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:38 PM
can you double confirm
02:38
Prabhat
02:38 PM
and check its value
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:38 PM
Yes the secret is there
02:38
Jens
02:38 PM
The values are also there
02:39
Jens
02:39 PM
Thatโ€™s whatโ€™s puzzling me
02:40
Jens
02:40 PM
Does the image actually use a shell as entrypoint?
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:40 PM
can you check the pod definition, on how secret is being mounted,
02:40
Prabhat
02:40 PM
just to be sure
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:40 PM
Usually I encounter these problems with images that use the binary as entrypoint
02:40
Jens
02:40 PM
just a sec
02:41
Jens
02:41 PM
apiVersion: v1
kind: Pod
metadata:
  annotations:
    : eks.privileged
  creationTimestamp: "2023-05-25T14:33:04Z"
  generateName: zinc-zincobserve-ingester-
  labels:
    : zinc
    : zincobserve
    controller-revision-hash: zinc-zincobserve-ingester-78d9c59bd7
    role: ingester
    : zinc-zincobserve-ingester-0
  name: zinc-zincobserve-ingester-0
  namespace: zinc
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: zinc-zincobserve-ingester
    uid: 09c146df-4003-4477-a841-b621ee66fbe1
  resourceVersion: "1018826688"
  uid: 8d4976be-e1b3-4b35-8703-02905c20f391
spec:
  containers:
  - env:
    - name: ZO_NODE_ROLE
      value: ingester
    envFrom:
    - secretRef:
        name: zinc-zincobserve
    - configMapRef:
        name: zinc-zincobserve
    image: public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-23b3a0e-amd64
    imagePullPolicy: IfNotPresent
    name: zincobserve
    ports:
    - containerPort: 5080
      name: http
      protocol: TCP
    - containerPort: 5081
      name: grpc
      protocol: TCP
    resources: {}
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /data
      name: data
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-58ptl
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: zinc-zincobserve-ingester-0
  nodeName: ip-172-16-177-133.eu-central-1.compute.internal
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext:
    fsGroup: 2000
    runAsGroup: 3000
    runAsNonRoot: true
    runAsUser: 10000
  serviceAccount: zinc-zincobserve
  serviceAccountName: zinc-zincobserve
  subdomain: zinc-zincobserve
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: 
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: 
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-zinc-zincobserve-ingester-0
  - name: kube-api-access-58ptl
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-05-25T14:33:04Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-05-25T14:33:30Z"
    message: 'containers with unready status: [zincobserve]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-05-25T14:33:30Z"
    message: 'containers with unready status: [zincobserve]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-05-25T14:33:04Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: b
    image: public.ecr.aws/zinclabs/zincobserve-dev:v0.4.3-23b3a0e-amd64
    imageID: public.ecr.aws/zinclabs/[email protected]:fa2b5c209fe61e882bb0f3c6c7888fb4f89b576d048c0a69e8282c1bbceba49e
    lastState:
      terminated:
        containerID: b
        exitCode: 1
        finishedAt: "2023-05-25T14:39:09Z"
        reason: Error
        startedAt: "2023-05-25T14:39:09Z"
    name: zincobserve
    ready: false
    restartCount: 6
    started: false
    state:
      waiting:
        message: back-off 5m0s restarting failed container=zincobserve pod=zinc-zincobserve-ingester-0_zinc(8d4976be-e1b3-4b35-8703-02905c20f391)
        reason: CrashLoopBackOff
  hostIP: 172.16.177.133
  phase: Running
  podIP: 172.16.173.116
  podIPs:
  - ip: 172.16.173.116
  qosClass: BestEffort
  startTime: "2023-05-25T14:33:04Z"
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:42 PM
we use CMD ["/zincobserve"]
02:42
Prabhat
02:42 PM
checking pod def
02:43
Prabhat
02:43 PM
All looks fine
02:44
Prabhat
02:44 PM
we have secret, and mapping is correct and pod has started.
02:45
Prabhat
02:45 PM
There isn't really anything more than that when it comes to setting up an env variable
02:45
Prabhat
02:45 PM
๐Ÿค”
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:46 PM
Turned on the full backtrace , now I have a list of 26 <unknown> ๐Ÿ˜„ which are not really helping
๐Ÿคฃ 1
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:46 PM
can you try deleting the chart. and also deleting the PVCs and reinstall.
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:47 PM
Iโ€™ll try that
02:53
Jens
02:53 PM
I just inspected the docker image:

 "Cmd": [
   "/zincobserve"
 ],
 "Entrypoint": null,
02:53
Jens
02:53 PM
It does not use an entrypoint so the rust process needs to handle the environment
02:54
Prabhat
02:54 PM
this is the source
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:54 PM
I only have very superficial knowledge of Rust, so IDK if it handles env in containers
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:54 PM
Actually we have been running this in EKS and GKE for months
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:56 PM
Iโ€™ll just try and overwrite the entrypoint in the STS/Deployment to test
02:57
Jens
02:57 PM
Iโ€™ll have to hop on a meeting in a few minutes so Iโ€™ll report my findings later/tomorrow
๐Ÿ‘ 1
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:57 PM
STS ?
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
02:57 PM
stateful set
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
02:57 PM
ohk
02:59
Prabhat
02:59 PM
Also if you can generate the manifests using this command and see if it looks right.
02:59
Prabhat
02:59 PM
helm --namespace zinc -f values.yaml template zo1 zinc/zincobserve > zo1.yaml
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:01 PM
Tried changing the entrypoint - to no avail
04:01
Jens
04:01 PM
butโ€ฆ
04:06
Jens
04:06 PM
<mailto:[email protected]|[email protected]>
apparently fails the email regex validation, which does not permit - before the @
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:07 PM
interesting.
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:07 PM
THere is only on eerror message for all three conditions
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:08 PM
can you test [email protected]
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:08 PM
or anything without -
04:08
Prabhat
04:08 PM
I think we are validating it as well
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:08 PM
A different matches the regex
04:09
Jens
04:09 PM
The current regex is overly restrictive IMO (and it is impossible to validate all valid email formats using regex anyway)
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:10 PM
you are right
04:11
Prabhat
04:11 PM
if !db::user::root_user_exists().await {
        if CONFIG.auth.root_user_email.is_empty()
            || !email_regex.is_match(&CONFIG.auth.root_user_email)
            || CONFIG.auth.root_user_password.is_empty()
        {
            panic!("Please set root user email-id & password using ZO_ROOT_USER_EMAIL & ZO_ROOT_USER_PASSWORD enviornment variables");
        }
        let
04:11
Prabhat
04:11 PM
We do test it
04:11
Prabhat
04:11 PM
can you try with your email id to confirm this
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:11 PM
At least we found out that it is not k8s that is broken ๐Ÿ˜„
04:11
Jens
04:11 PM
Iโ€™ll try
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:11 PM
haha. yes
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:20 PM
Ok, hit another issue with S3 configuration: thread 'tokio-runtime-worker' panicked at 's3 init config error: Generic { store: "S3", source: MissingRegion }', src/infra/storage/remote.rs:231:17

but that is a different issue. (I saw some of the S3 related env vars being empty or containing apparent default values)
04:20
Jens
04:20 PM
Butr changing the email resolved the ROOT_USER env issues
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:22 PM
just set this value
ZO_S3_REGION_NAME
04:22
Prabhat
04:22 PM
and you should be fine
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:23 PM
The bucket name is also wrong. Iโ€™ll take a look
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:24 PM
also bucket name should be in config and not auth
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:26 PM
The name is in the config map not the secret, but mysuperduperbucket looks like a dummy value
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:28 PM
yes, its a dummuy value
04:29
Prabhat
04:29 PM
I saw in your initial config that you gave
04:29
Prabhat
04:29 PM
[email protected]***.com
  ZO_ROOT_USER_PASSWORD: YP...r
  ZO_S3_ACCESS_KEY: AKI...
  ZO_S3_BUCKET_NAME: zinc-20230525135027032600000002
  ZO_S3_SECRET_KEY: O...
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:29 PM
Yes, but helm just drops unknown values silently
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
04:30 PM
right
04:59
Prabhat
04:59 PM
is it working now?
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
04:59 PM
Yes
๐ŸŽ‰ 2
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:00 PM
do use the dev image. It looks stable for now and has way more features than release image. We will be releasing a new version soon.
๐Ÿ‘ 1
05:01
Prabhat
05:01 PM
BTW, what is your use case? Logs, metrics or traces?
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
05:02 PM
Right now I am looking for a more lightweight alternative for logs than ES/AWS OSS
05:02
Jens
05:02 PM
I also have prometheus deployed so maybe Iโ€™ll also try if we can integrate our metrics
05:03
Jens
05:03 PM
The appllications currently do not use OTEL (yet?)
05:04
Jens
05:04 PM
I am trying to set up logging-operator to shift logging configuration to teams - the http output should work well with zincobserve
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:04 PM
cool.
05:04
Prabhat
05:04 PM
Would love to know how things progress.
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
05:06 PM
Iโ€™ll keep you updated. We are also setting up Loki in parallel to compare
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:06 PM
cool. We are still building PromQL support on metrics. So expect some rough edges and missing features there.
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
05:06 PM
(although feature sets may not be actually comparable)
05:06
Jens
05:06 PM
My primary problem right now is logs, metrics second ๐Ÿ™‚
๐Ÿ‘ 1
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:07 PM
yeah. Loki is generally good. The challenge that I have heard with Loki is around high cardinality data.
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
05:08 PM
high cardinality is everybodys boss fight - prometheus is also great at inflating to stellar resource consumption with high label cardinality
05:09
Jens
05:09 PM
Already pairing it with thanos, so prometheus only has to keep 2h worth of metrics
๐Ÿ‘ 1
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:10 PM
ZincObserve is much better at handling high cardinality data due to parquet storage format
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
05:10 PM
Iโ€™m looking forward to take it for a spin ๐Ÿ™‚
Prabhat
Photo of md5-23052f31f8f3c4b1bb3297fbc3a2aec5
Prabhat
05:11 PM
Looking forward to hearing more from you. Have a good day/night ahead.
Jens
Photo of md5-0c6460d8824800dbbd50c6ff1148f9f1
Jens
05:12 PM
Have a good night. Iโ€™m off for a walk with the dog ๐Ÿ™‚
๐Ÿ‘ 1