While creating a cluster using the steps in Create an Azure Data Explorer cluster and database, in the Configurations tab, select Streaming ingestion > On. You can enable the capability when creating the cluster, or add it to an existing cluster. SSD capacity: Enabling streaming ingestion on a cluster, even when data isn't ingested via streaming, uses part of the local SSD disk of the cluster machines for streaming ingestion data and reduces the storage available for hot cache.Įnable streaming ingestion on your clusterīefore you can use streaming ingestion, you must enable the capability on your cluster and define a streaming ingestion policy.For more information see Streaming ingestion and schema changes. Schema updates: Schema updates, such as creation and modification of tables and ingestion mappings, may take up to five minutes for the streaming ingestion service.This includes any data created for update policies during the ingestion. Data size limit: The data size limit for a streaming ingestion request is 4 MB.For two core SKUs, such as D11, the maximal supported load is 12 concurrent ingestion requests. For example, for 16 core SKUs, such as D14 and L16, the maximal supported load is 96 concurrent ingestion requests. The number of concurrent ingestion requests is limited to six per core. VM and cluster size: Streaming ingestion performance and capacity scales with increased VM and cluster sizes.The main contributors that can impact streaming ingestion are: Performance and operational considerations If you are using C# for your custom application, you may find it more convenient using the programmatic approach. You can manage the process to enable and disable streaming ingestion on your cluster using the Azure portal or programmatically in C#. High development overhead to create an application ingest the data, handle errors, and ensure data consistency Use the following table to help you choose the ingestion type that's appropriate for your environment: Criterionĭata delay between ingestion initiation and the data available for queryįast and easy setup, no development overhead You may also find the C# streaming ingestion sample application helpful. Use the information in this topic to configure custom ingestion. For information about setting these up, see Event Hub or IoT Hub data ingestion methods.Ĭustom ingestion requires you to write an application that uses one of the Azure Data Explorer client libraries. ![]() Hubs are configured as table streaming data sources. Two streaming ingestion types are supported: Ingestion type Choose the appropriate streaming ingestion type To learn more about different ingestion methods, see data ingestion overview. If the stream of data into each table is high (over 4 GB per hour), consider using batch ingestion.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |