QTag: Turbocharge Your SQL Comments
Here's a secret: your tools are talking to you.
Most analytics tools today automatically insert metadata into each query they submit. They do this by adding a comment into each query along with various metadata. That metadata can be very useful when trying to better understand what is going on with your database or data warehouse.
QTag is set of open source tools & specification to make that metadata easily accessible.
Check out the tools on Github or in the Snowflake Marketplace
select * from events
-- {"app": "dbt"}
A QTag is simply a comment in a SQL query that includes additional structured metadata about the query. QTags can be inserted by a tool or a by a user. A single query can have many qtags and qtag tooling automatically extracts that information for analysis. There are two main types of QTags: defined qtags and self-describing qtags.
Defined QTags
Defined QTags are comment patterns that are well-known and have been added explicitly to the QTag tools. These describe how to identify the unique comment signature of each tool along with a categorization of each type of metadata (to ease future metrics/reporting of the metadata).
Self-describing QTags
Since QTags are built on top of JSON, you can have your own comments be automatically extracted by QTag tools by following a few simple rules. This will allow you to take advantage of automatic metadata categories if your tools are private. Self-describing QTags are of the format:
-- qtag {"key:type":"value"}
QTags are Categorized
Each type of QTag attribute is categorized asone of the following attribute types: a dimension, a measure, a user identity, an application identifier, or a trace. This enables easy automated reporting and analysis.
Odds are, you already have QTags in your queries
Since many tools automatically inject comments into queries they submit, chances are you have already have QTags in your queries. The first step you can take with QTags is understand the data you already have available to you. From there, you can enhance your common SQL development patterns to better utilize QTags in your workloads. The QTag OSS project has several ways to get started.
JavaScript Extractor
The QTag Github repo has a high-performance JavaScript implementation of the QTag extractor. This can be used in your existing applications or most databases as a user-defined JavaScript function.
QTag for Snowflake
If you use Snowflake, you can easily enable QTag as a collection of Snowflake functions through the freely available DeckTools Snowflake Marketplace Listing.
We hope QTag can make things a little better for you
QTag was brought to you with love by the people at Sundeck. It's an open source project licensed under the Apache2 license. We built it when trying to help our users better understand and influence their Snowflake Consumption. We hope you find it useful!
Declare the QTag for your favorite tool
We're always looking to add to the collection of pre-defined tools QTag automatically identifies. Add your favorite to those supported including dbt, Mode, Sigma and Hex. Open a PR for your favorite tool.
Add More Test Cases
We're always looking to improve QTag. If you found a bug (or have a gnarly scenario where you want to make sure QTag works for you, feel free to open a PR for additional test cases.
Contribute New Features
Think QTag could be improved? Let us know how. We're always looking for new ideas. If you have one, feel free to share (or better yet, post a PR!)