Skip to content

Reverse-engineering the MapR ticket format

How I reverse-engineered the MapR ticket format, recovered its protobuf definition and parsed tickets in Python, using only public artifacts.

On this page

The Kubernetes clusters I worked on mounted MapR volumes with several hundred tickets, each stored in a Secret and each valid for a limited time. When one of them expired, a volume stopped mounting, and the only clue was a vague event from the CSI driver. I wanted to read a ticket’s expiry date without installing the MapR client, so I took the format apart.

Short version: a MapR ticket is AES-GCM encrypted with a hardcoded key, 32 bytes of the letter A. The plaintext is a protobuf message whose definition ships inside the public maprfs jar. Decrypt, parse it with the recovered .proto, and you get the user, groups and expiry time with no MapR software installed.

Everything below comes from publicly available artifacts, and the code snippets are trimmed.

Why expired tickets hurt

MapR, now sold as HPE Data Fabric, bundles a distributed file system, a NoSQL database and a streaming engine on top of the Hadoop stack. The file system, MapR-FS, can back Kubernetes volumes through the MapR CSI driver. The driver has a GitHub repository, but it only holds manifests, Helm charts and Dockerfiles for prebuilt images, no source code.

Access to MapR-FS needs a MapR ticket: a Base64-encoded binary blob with credentials and a limited lifetime. You create one with maprlogin on a machine with the MapR client. In Kubernetes, the ticket goes into a Secret that the PersistentVolume references:

mapr-pv-example.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mapr-ticket-example
  namespace: default
type: Opaque
data:
  CONTAINER_TICKET: ZGVtby5tYXByLmNvbSArQ3plK3F3WUNiQVhHYno1Nk9PN1VGK2xHcUwzV1BYck5rTzFTTGF3RUVEbVNiZ05sMDE5eEJlQlkza3ZoK1IxM2l6L21DbndwenNMUXc0WTVqRW52NUd0dUlXYmVvQzk1aGE4VKwX8MKcE6Kn9nZ2AF0QminkHwNVBx6TDriGZffyJCfZzivBwBSdKoQEWhBOPFCIMAi7w2zV/SX5Ut7u4qIKvEpr0JHV7sLMWYLhYncM6CKMd7iECGvECsBvEZRVj+dpbEY0BaRN/W54/7wNWaSVELUF6JWHQ8dmsqty4cZlI0/MV10HZzIbl9sMLFQ=
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mapr-pv-example
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Delete
  capacity:
    storage: 5Gi
  csi:
    nodePublishSecretRef:
      name: "mapr-ticket-example"
      namespace: "default"
    driver: com.mapr.csi-kdf
    volumeHandle: mapr-pv-example
    volumeAttributes:
      volumePath: "/"
      cluster: "demo.mapr.com"
      cldbHosts: "10.10.102.96"
      securityType: "secure"

A ticket lives for at most 30 days by default, so with several hundred of them something is always about to expire. When one does, the volume stops mounting, and all you get is this Kubernetes event:

Failed to start fuse process. Check cluster name and user ticket(if secure) specified

The driver doesn’t expose any ticket metadata. The only MapR tool that reads a ticket is maprlogin print, and that needs the full MapR client.

Find the code behind maprlogin

The format isn’t documented, so I started with maprlogin itself. Its print subcommand shows the basic metadata:

Terminal window
$ maprlogin print -ticketfile /tmp/maprticket_1000
Opening keyfile /tmp/maprticket_1000
my.cluster.com: user = juser, created = 'Mon Sep 17 08:30:26 PDT 2018', expires = 'Mon Oct 01 08:30:26 PDT 2018', RenewalTill = 'Wed Oct 17 08:30:26 PDT 2018', uid = 20001, gids = 54261, CanImpersonate = false

maprlogin turns out to be a shell script around a Java class:

Terminal window
"$JAVA_HOME"/bin/java ${MAPR_COMMON_JAVA_OPTS} ${MAPRLOGIN_SUPPORT_OPTS} \
    -classpath ${MAPRLOGIN_CLASSPATH}\
    ${MAPRLOGIN_OPTS} com.mapr.login.MapRLogin $args

To find the jar with MapRLogin in it, I used a search based on this StackOverflow answer:

Terminal window
$ find . -name '*.jar' -print0 | \
    xargs -0 -I '{}' sh -c 'jar tf {} | grep com.mapr.login.MapRLogin && echo {}'
com/mapr/login/MapRLogin.class
com/mapr/login/MapRLoginException.class
./maprfs-7.5.0.0-mapr.jar

All of that needs a MapR client installation. The jars are also on HPE’s public Maven repository at repository.mapr.com, though, so I downloaded version 7.5.0.0 of maprfs directly.

Decompile the jar

Procyon turns the jar back into Java:

Terminal window
brew install procyon-decompiler
Terminal window
$ procyon-decompiler -jar ./maprfs-7.5.0.0-mapr.jar -o mapr
Decompiling com/mapr/baseutils/BaseUtilsHelper...
Decompiling com/mapr/baseutils/BinaryString...
[...]
Decompiling com/mapr/login/MapRLogin...
[...]

The output is very readable Java.

Follow the print command

MapRLogin.execute dispatches print like this:

if (command.equals("print")) {
    handlePrint(inTicketFile, type);
    return;
}

handlePrint eventually calls:

final Security.TicketAndKey tk =
    com.mapr.security.Security.GetTicketAndKeyForCluster(sKType, cluster2, err);
if (tk != null) {
    printTicket(cluster2, tk);
}

GetTicketAndKeyForCluster hands off to a JNI method with no implementation in the jar, which was a dead end. The decompiled code also has ClientSecurity.getTicketAndKeyForCluster, though, and that one is implemented:

decryptedTicketAndKeyStream =
    this.decodeDataFromKeyFile(encryptedClientTicketAndKey);

decodeDataFromKeyFile is short:

private byte[] decodeDataFromKeyFile(final String encodedData) {
    final byte[] key = this.getKeyForKeyFile();
    final byte[] decryptedData = this.aesDecrypt(key, encryptedData);
    return decryptedData;
}

AES decryption. For a moment I thought that was the end of it, until I looked at where the key comes from:

static ClientSecurity.KEY_SIZE_IN_BYTES = 32;


private byte[] getKeyForKeyFile() {
    final byte[] keybuf = new byte[ClientSecurity.KEY_SIZE_IN_BYTES];
    for (int i = 0; i < ClientSecurity.KEY_SIZE_IN_BYTES; ++i) {
        keybuf[i] = 65;
    }
    return keybuf;
}

The key is 32 bytes of 0x41, the letter A, hardcoded and the same for every ticket. aesDecrypt uses AES-GCM, which is a reasonable cipher. The key management is another matter.

Decrypt a ticket

With the key known, decryption in Python is a few lines:

decrypt.py
import sys
from base64 import b64decode


from cryptography.hazmat.primitives.ciphers.aead import AESGCM




def aes_decrypt(key: bytes, cipher_text: bytes) -> bytes:
    iv = cipher_text[:16]
    aesgcm = AESGCM(key)
    plain_text = aesgcm.decrypt(iv, cipher_text[16:], None)
    return plain_text




if __name__ == "__main__":
    ticket = sys.stdin.read()
    host, secret = ticket.split(" ")


    key: bytes = ("A" * 32).encode()
    cipher_text = b64decode(secret)
    decrypted_data = aes_decrypt(key, cipher_text)


    print(decrypted_data)

I tested it with MapR tickets I found in public repositories:

test-tickets.txt
demo.mapr.com +Cze+qwYCbAXGbz56OO7UF+lGqL3WPXrNkO1SLawEEDmSbgNl019xBeBY3kvh+R13iz/mCnwpzsLQw4Y5jEnv5GtuIWbeoC95ha8VKwX8MKcE6Kn9nZ2AF0QminkHwNVBx6TDriGZffyJCfZzivBwBSdKoQEWhBOPFCIMAi7w2zV/SX5Ut7u4qIKvEpr0JHV7sLMWYLhYncM6CKMd7iECGvECsBvEZRVj+dpbEY0BaRN/W54/7wNWaSVELUF6JWHQ8dmsqty4cZlI0/MV10HZzIbl9sMLFQ=
demo.mapr.com cj1FDarNNKh7f+hL5ho1m32RzYyHPKuGIPJzE/CkUqEfcTGEP4YJuFlTsBmHuifI5LvNob/Y4xmDsrz9OxrBnhly/0g9xAs5ApZWNY8Rcab8q70IBYIbpu7xsBBTAiVRyLJkAtGFXNn104BB0AsS55GbQFUN9NAiWLzZY3/X1ITfGfDEGaYbWWTb1LGx6C0Jjgnr7TzXv1GqwiASbcUQCXOx4inguwMneYt9KhOp89smw6GBKP064DfIMHHR6lgv0XhBP6d9FVJ1QWKvcccvi2F3LReBtqA=
demo.mapr.com IGem6fUksZ1pd4iut978SKElS4ktecRsAkrl+qwPYc7xhfMg4wkwALKDmFmpc8Xvrm1L9Et0jVBoyhCWMDCjhToZ8b6FsfCn8wdCOB0MWm9CRobGv7MDsoEO2TQ5Bnh8i/VfuthKFxd3Om9iZPVCI4I1S9h4p/77Al1GzTGcfFFf1g9fq1HXftT9TEDyLdABIyATJbzv8zD10IDT8P1f8nxl7lgT/7ZhGz7N24vSz6jBxHE7oHmvHzjW22xJwt7TJgvrP21boH9HTsTPiKZOpQMZ4zFo6JA4aNVlQQ0=

The output is still binary, but the string mapr shows up near the end, so the decryption works.

Find the serialization format

Back in the decompiled code, the imports give the format away:

import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.ByteString;
import com.mapr.fs.proto.Security;

The generated Security class is over 16,000 lines long. It contains a descriptorData variable, which is the original .proto definition embedded as a serialized binary string:

final String[] descriptorData = {
    "\n\u000esecurity.proto\u0012\u0007mapr.fs\"¬\u0002\n\u000eCredentialsMsg ... "
};

Extract the proto definition

The protobuf Python library can parse that descriptor into a binary FileDescriptorProto:

rebuild_textproto.py
import sys
import re


import google.protobuf.descriptor_pb2 as descriptor_pb2


for line in sys.stdin.buffer:
    if b"security.proto" in line:
        break


m = re.search(r"^.*=\s*\{\s*\"(.*)\"\s*\}.*$", line.decode("utf-8"))
if m:
    data = m.group(1)
else:
    raise Exception("Could not find the string between `= { ... }`")


# fix encoding — hacky but functional
data = data.encode("latin-1").decode("unicode-escape").encode("latin-1")


fds = descriptor_pb2.FileDescriptorSet()
fds.file.append(descriptor_pb2.FileDescriptorProto())
fds.file[0].ParseFromString(data)
serialized_data = fds.file[0].SerializeToString()
sys.stdout.buffer.write(serialized_data)

That gives another binary blob. To turn a FileDescriptorProto back into a readable .proto file, you need DebugString(), and only the C++ protobuf library has it (per this StackOverflow answer).

Render it with C++

fds2proto.cpp
#include <google/protobuf/descriptor.h>
#include <google/protobuf/descriptor.pb.h>
#include <iostream>


int main()
{
  google::protobuf::FileDescriptorProto fileProto;


  if (!fileProto.ParseFromIstream(&std::cin))
  {
    std::cerr << "Failed to parse FileDescriptorProto from stdin" << std::endl;
    return 1;
  }


  google::protobuf::DescriptorPool pool;
  const google::protobuf::FileDescriptor* desc = pool.BuildFile(fileProto);
  std::cout << desc->DebugString() << std::endl;


  return 0;
}

Compile it:

Terminal window
brew install protobuf
export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib
export LD_LIBRARY_PATH=/opt/homebrew/lib:$LD_LIBRARY_PATH
g++ -o fds2proto fds2proto.cpp -std=c++17 -lprotobuf -pthread

Then chain everything together:

Terminal window
$ python rebuild_textproto.py < Security.java | ./fds2proto | tee security.proto
syntax = "proto2";


package mapr.fs;


option java_package = "com.mapr.fs.proto";
option optimize_for = LITE_RUNTIME;
option go_package = "ezmeral.hpe.com/datafab/fs/proto";


enum SecurityProg {
  ChallengeResponseProc = 1;
  RefreshTicketProc = 2;
}


[...]


message GetJwtTicketResponse {
  optional string error = 1;
  optional int32 status = 2;
  optional bytes maprTicket = 3;
}

That’s the .proto definition. It has every security-related message in the MapR codebase. Only the ticket messages matter here, but the full file works fine for code generation.

Parse a ticket end to end

Generate Python bindings from the recovered proto:

Terminal window
protoc --proto_path=./ --pyi_out=./ --python_out=./ security.proto

Then parse a ticket:

parse.py
import sys
from base64 import b64decode


from decrypt import aes_decrypt
from security_pb2 import TicketAndKey


ticket = sys.stdin.read()
host, secret = ticket.split(" ")


key: bytes = ("A" * 32).encode()
cipher_text = b64decode(secret)
decrypted_data = aes_decrypt(key, cipher_text)


ticket_and_key = TicketAndKey()
ticket_and_key.ParseFromString(decrypted_data)


print(ticket_and_key)

With the sample tickets:

Terminal window
$ python parse.py <<<"demo.mapr.com +Cze+qwYCbAXGbz56OO7UF+..."
encryptedTicket: "..."
userKey {
  key: "..."
}
userCreds {
  uid: 5000
  gids: 5000
  gids: 0
  gids: 5001
  userName: "mapr"
}
expiryTime: 922337203685477
creationTimeSec: 1522852297
maxRenewalDurationSec: 0


$ python parse.py <<<"demo.mapr.com cj1FDarNNKh7f+hL5ho1m3..."
encryptedTicket: "..."
userKey {
  key: "..."
}
userCreds {
  uid: 5000
  gids: 5000
  gids: 1000
  userName: "mapr"
}
expiryTime: 1550578429
creationTimeSec: 1549368829
maxRenewalDurationSec: 2592000
canUserImpersonate: true


$ python parse.py <<<"demo.mapr.com IGem6fUksZ1pd4iut978SK..."
encryptedTicket: "..."
userKey {
  key: "..."
}
userCreds {
  uid: 5000
  gids: 5000
  gids: 5003
  gids: 0
  userName: "mapr"
}
expiryTime: 1619735566
creationTimeSec: 1618525966
maxRenewalDurationSec: 2592000
canUserImpersonate: true
isExternal: true

That’s everything maprlogin print shows (user ID, group IDs, expiry, creation time, impersonation flags) without any MapR components installed.

This is what I wanted for the expiring Secrets. I built a Kubernetes controller on top of it that watches Secrets holding MapR tickets and annotates them with their expiry date. That controller is internal and not open source. What is public is mapr-ticket-parser, a Go module that decodes and encodes MapR tickets.

References

Published
Updated
Reading
4 min