Apache Atlas Hook & Bridge for Apache HBase

HBase Model

HBase model includes the following types:

  • Entity types:
    • hbase_namespace
      • super-types: !Asset
      • attributes: qualifiedName, name, description, owner, clusterName, parameters, createTime, modifiedTime
    • hbase_table
      • super-types: DataSet
      • attributes: qualifiedName, name, description, owner, namespace, column_families, uri, parameters, createtime, modifiedtime, maxfilesize, isReadOnly, isCompactionEnabled, isNormalizationEnabled, ReplicaPerRegion, Durability
    • hbase_column_family
      • super-types: DataSet
      • attributes: qualifiedName, name, description, owner, columns, createTime, bloomFilterType, compressionType, compactionCompressionType, encryptionType, inMemoryCompactionPolicy, keepDeletedCells, maxversions, minVersions, datablockEncoding, storagePolicy, ttl, blockCachedEnabled, cacheBloomsOnWrite, cacheDataOnWrite, evictBlocksOnClose, prefetchBlocksOnOpen, newVersionsBehavior, isMobEnabled, mobCompactPartitionPolicy

HBase entities are created and de-duped in Atlas using unique attribute qualifiedName, whose value should be formatted as detailed below. Note that namespaceName, tableName and columnFamilyName should be in lower case.

   hbase_namespace.qualifiedName:      <namespaceName>@<clusterName>
   hbase_table.qualifiedName:          <namespaceName>:<tableName>@<clusterName>
   hbase_column_family.qualifiedName:  <namespaceName>:<tableName>.<columnFamilyName>@<clusterName>

HBase Hook

Atlas HBase hook registers with HBase master as a co-processor. On detecting changes to HBase namespaces/tables/column-families, Atlas hook updates the metadata in Atlas via Kafka notifications. Follow the instructions below to setup Atlas hook in HBase:

  • Register Atlas hook in hbase-site.xml by adding the following:

  • untar apache-atlas-${project.version}-hbase-hook.tar.gz
  • cd apache-atlas-hbase-hook-${project.version}
  • Copy entire contents of folder apache-atlas-hbase-hook-${project.version}/hook/hbase to <atlas package>/hook/hbase
  • Link Atlas hook jars in HBase classpath - 'ln -s <atlas package>/hook/hbase/* <hbase-home>/lib/'
  • Copy <atlas-conf>/atlas-application.properties to the HBase conf directory.

The following properties in atlas-application.properties control the thread pool and notification details:

atlas.hook.hbase.synchronous=false # whether to run the hook synchronously. false recommended to avoid delays in HBase operations. Default: false
atlas.hook.hbase.numRetries=3      # number of retries for notification failure. Default: 3
atlas.hook.hbase.queueSize=10000   # queue size for the threadpool. Default: 10000

atlas.cluster.name=primary # clusterName to use in qualifiedName of entities. Default: primary

atlas.kafka.zookeeper.connect=                    # Zookeeper connect URL for Kafka. Example: localhost:2181
atlas.kafka.zookeeper.connection.timeout.ms=30000 # Zookeeper connection timeout. Default: 30000
atlas.kafka.zookeeper.session.timeout.ms=60000    # Zookeeper session timeout. Default: 60000
atlas.kafka.zookeeper.sync.time.ms=20             # Zookeeper sync time. Default: 20

Other configurations for Kafka notification producer can be specified by prefixing the configuration name with "atlas.kafka.". For list of configuration supported by Kafka producer, please refer to Kafka Producer Configs


  • Only the namespace, table and column-family create/update/ delete operations are captured by Atlas HBase hook. Changes to columns are be captured.

Importing HBase Metadata

Apache Atlas provides a command-line utility, import-hbase.sh, to import metadata of Apache HBase namespaces and tables into Apache Atlas. This utility can be used to initialize Apache Atlas with namespaces/tables present in a Apache HBase cluster. This utility supports importing metadata of a specific table, tables in a specific namespace or all tables.

Usage 1: <atlas package>/hook-bin/import-hbase.sh
Usage 2: <atlas package>/hook-bin/import-hbase.sh [-n <namespace regex> OR --namespace <namespace regex>] [-t <table regex> OR --table <table regex>]
Usage 3: <atlas package>/hook-bin/import-hbase.sh [-f <filename>]
           File Format: