Sqoop Model
The default Sqoop modelling is available in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. It defines the following types:
sqoop_operation_type(EnumType) - values [IMPORT, EXPORT, EVAL] sqoop_dbstore_usage(EnumType) - values [TABLE, QUERY, PROCEDURE, OTHER] sqoop_process(ClassType) - super types [Process] - attributes [name, operation, dbStore, hiveTable, commandlineOpts, startTime, endTime, userName] sqoop_dbdatastore(ClassType) - super types [DataSet] - attributes [name, dbStoreType, storeUse, storeUri, source, description, ownerName]
The entities are created and de-duped using unique qualified name. They provide namespace and can be used for querying as well: sqoop_process - attribute name - sqoop-dbStoreType-storeUri-endTime sqoop_dbdatastore - attribute name - dbStoreType-connectorUrl-source
Sqoop Hook
Sqoop added a SqoopJobDataPublisher that publishes data to Atlas after completion of import Job. Today, only hiveImport is supported in sqoopHook. This is used to add entities in Atlas using the model defined in org.apache.atlas.sqoop.model.SqoopDataModelGenerator. Follow these instructions in your sqoop set-up to add sqoop hook for Atlas in <sqoop-conf>/sqoop-site.xml:
- Sqoop Job publisher class. Currently only one publishing class is supported
- Atlas cluster name
- Copy <atlas-conf>/atlas-application.properties to to the sqoop conf directory <sqoop-conf>/
- Link <atlas-home>/hook/sqoop/*.jar in sqoop lib
Refer Configuration for notification related configurations