Google has announced the general availability of change streams, which allows users to track and stream out changes (inserts, updates, and deletes) from their Cloud Spanner database in near real-time.
Cloud Spanner change streams are available to all customers at no additional cost. One will have to pay only for any extra compute and storage of the change stream data at the regular Spanner rates.
Spanner change streams are created with DDL, similar to creating tables and indexes. Change stream DDL requires the same IAM permission as any other schema change. A change stream can track changes on a set of columns, a set of tables, or an entire database. Each change stream can have a retention period of anywhere from one day to seven days, and you can set up multiple change streams to track exactly what you need for your specific objectives. Spanner connector for Apache Beam allows building scalable data processing pipelines for Google Cloud Dataflow.
“Since change streams are built right into Spanner, there’s no software to install, and you get external consistency, industry-leading availability, and effortless scale with the rest of the database,” the company said.
Introduced at this year’s Google Data Cloud Summit, change streams let you capture change data from Spanner databases and easily integrate it with other systems to unlock new value. Everyday use cases include:
- Replicating Spanner data changes to a data warehouse, such as BigQuery, for analytics.
- Triggering application logic based on data changes sent to a message queue, such as Pub/Sub.
- Storing data changes in Cloud Storage for compliance or archival purposes.
Change streams are highly flexible and configurable, letting you capture change data for the exact data you care about and the exact time period that matters for businesses.
