{"_id":"57fcc4870312b20e00ac64e2","__v":0,"parentDoc":null,"project":"5435687035740020002a1c04","version":{"_id":"57fcc4860312b20e00ac64c0","project":"5435687035740020002a1c04","__v":1,"createdAt":"2016-10-11T10:52:54.637Z","releaseDate":"2016-10-11T10:52:54.637Z","categories":["57fcc4860312b20e00ac64c1","57fcc4860312b20e00ac64c2","57fcc4860312b20e00ac64c3","57fcc4860312b20e00ac64c4","57fcc4860312b20e00ac64c5","57fcc4860312b20e00ac64c6","57fcc4860312b20e00ac64c7","57fcc4860312b20e00ac64c8","57fcc4860312b20e00ac64c9","57fcc4860312b20e00ac64ca","57fcc4860312b20e00ac64cb"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"[APP-1265], [APP-1035]","version_clean":"5.2.0","version":"5.2"},"category":{"_id":"57fcc4860312b20e00ac64c8","__v":0,"version":"57fcc4860312b20e00ac64c0","project":"5435687035740020002a1c04","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2015-11-04T23:13:08.376Z","from_sync":false,"order":7,"slug":"lua-scripting","title":"Agent Lua Scripting"},"user":"5589fde775eaf50d004e4b0c","updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-11-06T00:24:24.058Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":4,"body":"Data series are sequences of data points, each composed of a floating-point numeric value and a timestamp, which also serves as a sorting key. The Agent adds functionality to Lua that allows you to retrieve data series from the Agent's data management system and manipulate them to alter their contents, make decisions, and populate flows.\n\nThe importing the `telemetry/storage` library exposes the `series()` function which returns a reference to a data series expression:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\n\\nlocal series = st.series(\\\"test\\\")\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\nIt takes a single parameter:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Type\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`name`\",\n    \"0-1\": \"string\",\n    \"0-2\": \"The name of the series (required).\"\n  },\n  \"cols\": 3,\n  \"rows\": 1\n}\n[/block]\nThe name of a series must start with a letter and can contain letters, numbers, underscores, and periods.\n\n## Types of series expression results\n\nWhen interrogating a series—for example, running an aggregation, or extracting a value, the system returns results in one of two formats.\n\nSingle-value formats are represented by an expression that, used on its own, returns the value associated with a datapoint. The expression also exposes a `timestamp` property that returns the corresponding timestamp, and a `value` property that can be used to forcefully return the data points value. For example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\nlocal value = series.last()\\n\\n-- Outputs the value to a number flow directly\\noutput.number = {\\n  value = value.value\\n}\\n\\n-- Request the timestamp instead\\nif value.ts < os.time() - 10000 then\\nend\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\nAggregate results that contain more than one data point are returned as an expression that, on its own, returns an array of each of the data points. Each data point contains a table which holds the `ts` property that contains the timestamp associated with the result, and a `value` property that contains the actual value of the data point. You can also use the `values()` and `ts()` functions to retrieve an array containing only values or only timestamps:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\nlocal value = series.aggregate(st.Functions.AVG, \\\"1m\\\", 10)\\n\\n-- Get the values only\\nlocal values = value.values()\\n\\n-- Get the timestamps only\\nlocal timestamps = value.ts()\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n## Retrieving the last value of a series\n\nThe `last()` function of a series expression returns the last value stored in the series. This function takes no arguments and does not alter the contents of the series.\n\nFor example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\nlocal value = series.last()\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n## Popping the last value out of a series\n\nYou can also retrieve the last value of a series using the `pop()` function. Unlike `last()`, this will also _remove_ the value from the series.\n\nFor example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\nlocal value = series.pop()\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"Popping from an empty series is not allowed\",\n  \"body\": \"It is a programmer error to pop a value out of an empty series. Use the `compute()` function's `st.Functions.COUNT` option to ensure that there are values to be popped out before calling `pop()`.\"\n}\n[/block]\n## Adding new values to a series\n\nYou can use the `push()` function to add new data points to a data series. The function takes two arguments:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Type\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`value`\",\n    \"0-1\": \"Number\",\n    \"0-2\": \"The value to be pushed (required).\",\n    \"1-0\": \"`timestamp`\",\n    \"1-1\": \"Number\",\n    \"1-2\": \"The UNIX timestamp associated with the data point (optional, defaults to the value of `os.time()`).\"\n  },\n  \"cols\": 3,\n  \"rows\": 2\n}\n[/block]\nPlease note that push does not return a value. Here's an example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\n-- Add a value for ten seconds ago\\nseries.push(45, (os.time() - 10))\\n\\n-- Add a value for \\\"now\\\"\\nseries.push(45)\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"Using a data series as a stack\",\n  \"body\": \"A data series can be used as a convenience stack for keeping track of state between subsequent runs of a job. Remember, however, that data series are sorted by the timestamps associated with their data points; therefore, `push()` and `pop()` can only be used to create and manage a stack if you always use incrementing timestamps when you add new values to a series.\\n\\nIn practice, omitting the `timestamp` argument when calling `push()` will accomplish this goal.\"\n}\n[/block]\n## Accessing series values directly\n\nSometimes you will want to retrieve the values of a series without any aggregation. The `items()` function allows for this. It takes a single argument representing the number of items that you want to retrieve. These items are pulled from the end of the array. For example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\n-- Pull the last 10 items from the series\\nlocal items = series.items(10)\\n\\n-- Get the values only\\nlocal values = items.values()\\n\\n-- Get the timestamps only\\nlocal timestamps = items.ts()\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n## Series calculation functions\n\nSeries have access to the `compute()` and `aggregate()` functions which expose a number of functions designed to perform operations on a subset of the series' contents:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Function\",\n    \"h-1\": \"Description\",\n    \"0-0\": \"`AVG`\",\n    \"0-1\": \"Compute the average of the values.\",\n    \"1-0\": \"`COUNT`\",\n    \"1-1\": \"Count the number of data points.\",\n    \"2-0\": \"`MIN`\",\n    \"2-1\": \"Compute the minimum value.\",\n    \"3-0\": \"`MAX`\",\n    \"3-1\": \"Compute the maximum value.\",\n    \"4-0\": \"`STDDEV`\",\n    \"4-1\": \"Compute the standard deviation of the values.\",\n    \"5-0\": \"`SUM`\",\n    \"5-1\": \"Compute the sum of the values.\"\n  },\n  \"cols\": 2,\n  \"rows\": 6\n}\n[/block]\n## Computing operations on a series\n\nThe `compute()` function takes three arguments and returns a floating-point numeric expression:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Type\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`func`\",\n    \"0-1\": \"constant\",\n    \"0-2\": \"Name of the series calculation function that will execute. This is accessed through the `Functions` table of the storage library. For example: `st.Functions.SUM` (required).\",\n    \"1-0\": \"`since`\",\n    \"1-2\": \"The time interval since now on which to perform the computation (required).\",\n    \"1-1\": \"string\"\n  },\n  \"cols\": 3,\n  \"rows\": 2\n}\n[/block]\nHere are some examples:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\n-- Compute the average over the last sixty minutes\\nlocal value = series.compute(st.Functions.AVG, \\\"60m\\\")\\n\\n-- Compute the standard deviation over the last thirty seconds\\nlocal value = series.compute(st.Functions.STDDEV, \\\"30s\\\")\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n## Aggregating data from a series\n\nThe `aggregate()` function allows you to perform an aggregation over the contents of the series, first grouping data by a given time period, then computing an operation of your choosing over each group.\n\nThe function takes three arguments:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Type\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`func`\",\n    \"0-1\": \"constant\",\n    \"0-2\": \"Name of the series calculation function that will execute. This is accessed through the `Functions` table of the storage library. For example: `st.Functions.SUM` (required).\",\n    \"1-0\": \"`interval`\",\n    \"1-1\": \"string\",\n    \"1-2\": \"An amount of time (in interval string format) by which data points must be grouped (required).\",\n    \"2-0\": \"`count`\",\n    \"2-1\": \"number\",\n    \"2-2\": \"The number of aggregate groups to be generated (required).\",\n    \"3-1\": \"number\",\n    \"3-2\": \"Timestamp for the maximum point in time that the aggregation will execute on before stopping. Defaults to the current time (optional).\",\n    \"3-0\": \"`max`\"\n  },\n  \"cols\": 3,\n  \"rows\": 4\n}\n[/block]\nThe `func` argument takes the name of one of the computation operations shown in the `Series calculation functions` section. For example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\n-- Get the 10-second average over the last 60 minutes\\nlocal value = series.aggregate(st.Functions.AVG, \\\"10s\\\", 60)\\n\\n-- Count the number of data points received every minute in the last day\\nlocal value = series.aggregate(st.Functions.COUNT, \\\"1m\\\", 1440)\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n## Trimming a data series\n\nIt's generally advisable to only store as much data in a data series as is strictly necessary for your specific purposes. Letting a series growing too large consumes disk space and could lead to slower operations.\n\nSeries expressions expose the `trimCount()` and `trimSince()` functions for this specific purpose.\n\n`trimCount()` takes one argument:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Type\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`count`\",\n    \"0-1\": \"number\",\n    \"0-2\": \"The maximum number of data points to retain.\"\n  },\n  \"cols\": 3,\n  \"rows\": 1\n}\n[/block]\n`trimSince()` takes _one_ argument:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Name\",\n    \"h-1\": \"Type\",\n    \"h-2\": \"Description\",\n    \"0-0\": \"`since`\",\n    \"0-1\": \"string\",\n    \"0-2\": \"The time interval since now for which data must be retained (required).\"\n  },\n  \"cols\": 3,\n  \"rows\": 1\n}\n[/block]\nFor example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"local st = require(\\\"telemetry/storage\\\")\\nlocal series = st.series(\\\"test\\\")\\n\\n-- Keep the last day's worth of data via interval\\nseries.trimSince(\\\"1d\\\")\\n\\n-- Keep at most the last 10,000 data points\\nseries.trimCount(10000)\",\n      \"language\": \"lua\"\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"warning\",\n  \"title\": \"Use `trim` functions with care!\",\n  \"body\": \"A call to `trimCount()` or `trimSince()` locks the entire Agent database while old rows are purged. The Agent compensates for this by queueing other operations (including the insertion of new data points) until after `trim` finishes executing.\\n\\nThis should not normally a problem, but under extreme loads, or when `trim` is executed too frequently, it could cause the insertion of data points out of sequence, affecting the results of querying operations until all data points have finished queueing into the database.\\n\\nTo minimize the chance of inconsistencies in the data, we recommend that you only call `trimCount()` or `trimSince()` when absolutely necessary.\"\n}\n[/block]","excerpt":"","slug":"accessing-series-data-from-lua","type":"basic","title":"Series Data"}
Data series are sequences of data points, each composed of a floating-point numeric value and a timestamp, which also serves as a sorting key. The Agent adds functionality to Lua that allows you to retrieve data series from the Agent's data management system and manipulate them to alter their contents, make decisions, and populate flows. The importing the `telemetry/storage` library exposes the `series()` function which returns a reference to a data series expression: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\n\nlocal series = st.series(\"test\")", "language": "lua" } ] } [/block] It takes a single parameter: [block:parameters] { "data": { "h-0": "Name", "h-1": "Type", "h-2": "Description", "0-0": "`name`", "0-1": "string", "0-2": "The name of the series (required)." }, "cols": 3, "rows": 1 } [/block] The name of a series must start with a letter and can contain letters, numbers, underscores, and periods. ## Types of series expression results When interrogating a series—for example, running an aggregation, or extracting a value, the system returns results in one of two formats. Single-value formats are represented by an expression that, used on its own, returns the value associated with a datapoint. The expression also exposes a `timestamp` property that returns the corresponding timestamp, and a `value` property that can be used to forcefully return the data points value. For example: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\nlocal value = series.last()\n\n-- Outputs the value to a number flow directly\noutput.number = {\n value = value.value\n}\n\n-- Request the timestamp instead\nif value.ts < os.time() - 10000 then\nend", "language": "lua" } ] } [/block] Aggregate results that contain more than one data point are returned as an expression that, on its own, returns an array of each of the data points. Each data point contains a table which holds the `ts` property that contains the timestamp associated with the result, and a `value` property that contains the actual value of the data point. You can also use the `values()` and `ts()` functions to retrieve an array containing only values or only timestamps: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\nlocal value = series.aggregate(st.Functions.AVG, \"1m\", 10)\n\n-- Get the values only\nlocal values = value.values()\n\n-- Get the timestamps only\nlocal timestamps = value.ts()", "language": "lua" } ] } [/block] ## Retrieving the last value of a series The `last()` function of a series expression returns the last value stored in the series. This function takes no arguments and does not alter the contents of the series. For example: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\nlocal value = series.last()", "language": "lua" } ] } [/block] ## Popping the last value out of a series You can also retrieve the last value of a series using the `pop()` function. Unlike `last()`, this will also _remove_ the value from the series. For example: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\nlocal value = series.pop()", "language": "lua" } ] } [/block] [block:callout] { "type": "warning", "title": "Popping from an empty series is not allowed", "body": "It is a programmer error to pop a value out of an empty series. Use the `compute()` function's `st.Functions.COUNT` option to ensure that there are values to be popped out before calling `pop()`." } [/block] ## Adding new values to a series You can use the `push()` function to add new data points to a data series. The function takes two arguments: [block:parameters] { "data": { "h-0": "Name", "h-1": "Type", "h-2": "Description", "0-0": "`value`", "0-1": "Number", "0-2": "The value to be pushed (required).", "1-0": "`timestamp`", "1-1": "Number", "1-2": "The UNIX timestamp associated with the data point (optional, defaults to the value of `os.time()`)." }, "cols": 3, "rows": 2 } [/block] Please note that push does not return a value. Here's an example: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\n-- Add a value for ten seconds ago\nseries.push(45, (os.time() - 10))\n\n-- Add a value for \"now\"\nseries.push(45)", "language": "lua" } ] } [/block] [block:callout] { "type": "info", "title": "Using a data series as a stack", "body": "A data series can be used as a convenience stack for keeping track of state between subsequent runs of a job. Remember, however, that data series are sorted by the timestamps associated with their data points; therefore, `push()` and `pop()` can only be used to create and manage a stack if you always use incrementing timestamps when you add new values to a series.\n\nIn practice, omitting the `timestamp` argument when calling `push()` will accomplish this goal." } [/block] ## Accessing series values directly Sometimes you will want to retrieve the values of a series without any aggregation. The `items()` function allows for this. It takes a single argument representing the number of items that you want to retrieve. These items are pulled from the end of the array. For example: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\n-- Pull the last 10 items from the series\nlocal items = series.items(10)\n\n-- Get the values only\nlocal values = items.values()\n\n-- Get the timestamps only\nlocal timestamps = items.ts()", "language": "lua" } ] } [/block] ## Series calculation functions Series have access to the `compute()` and `aggregate()` functions which expose a number of functions designed to perform operations on a subset of the series' contents: [block:parameters] { "data": { "h-0": "Function", "h-1": "Description", "0-0": "`AVG`", "0-1": "Compute the average of the values.", "1-0": "`COUNT`", "1-1": "Count the number of data points.", "2-0": "`MIN`", "2-1": "Compute the minimum value.", "3-0": "`MAX`", "3-1": "Compute the maximum value.", "4-0": "`STDDEV`", "4-1": "Compute the standard deviation of the values.", "5-0": "`SUM`", "5-1": "Compute the sum of the values." }, "cols": 2, "rows": 6 } [/block] ## Computing operations on a series The `compute()` function takes three arguments and returns a floating-point numeric expression: [block:parameters] { "data": { "h-0": "Name", "h-1": "Type", "h-2": "Description", "0-0": "`func`", "0-1": "constant", "0-2": "Name of the series calculation function that will execute. This is accessed through the `Functions` table of the storage library. For example: `st.Functions.SUM` (required).", "1-0": "`since`", "1-2": "The time interval since now on which to perform the computation (required).", "1-1": "string" }, "cols": 3, "rows": 2 } [/block] Here are some examples: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\n-- Compute the average over the last sixty minutes\nlocal value = series.compute(st.Functions.AVG, \"60m\")\n\n-- Compute the standard deviation over the last thirty seconds\nlocal value = series.compute(st.Functions.STDDEV, \"30s\")", "language": "lua" } ] } [/block] ## Aggregating data from a series The `aggregate()` function allows you to perform an aggregation over the contents of the series, first grouping data by a given time period, then computing an operation of your choosing over each group. The function takes three arguments: [block:parameters] { "data": { "h-0": "Name", "h-1": "Type", "h-2": "Description", "0-0": "`func`", "0-1": "constant", "0-2": "Name of the series calculation function that will execute. This is accessed through the `Functions` table of the storage library. For example: `st.Functions.SUM` (required).", "1-0": "`interval`", "1-1": "string", "1-2": "An amount of time (in interval string format) by which data points must be grouped (required).", "2-0": "`count`", "2-1": "number", "2-2": "The number of aggregate groups to be generated (required).", "3-1": "number", "3-2": "Timestamp for the maximum point in time that the aggregation will execute on before stopping. Defaults to the current time (optional).", "3-0": "`max`" }, "cols": 3, "rows": 4 } [/block] The `func` argument takes the name of one of the computation operations shown in the `Series calculation functions` section. For example: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\n-- Get the 10-second average over the last 60 minutes\nlocal value = series.aggregate(st.Functions.AVG, \"10s\", 60)\n\n-- Count the number of data points received every minute in the last day\nlocal value = series.aggregate(st.Functions.COUNT, \"1m\", 1440)", "language": "lua" } ] } [/block] ## Trimming a data series It's generally advisable to only store as much data in a data series as is strictly necessary for your specific purposes. Letting a series growing too large consumes disk space and could lead to slower operations. Series expressions expose the `trimCount()` and `trimSince()` functions for this specific purpose. `trimCount()` takes one argument: [block:parameters] { "data": { "h-0": "Name", "h-1": "Type", "h-2": "Description", "0-0": "`count`", "0-1": "number", "0-2": "The maximum number of data points to retain." }, "cols": 3, "rows": 1 } [/block] `trimSince()` takes _one_ argument: [block:parameters] { "data": { "h-0": "Name", "h-1": "Type", "h-2": "Description", "0-0": "`since`", "0-1": "string", "0-2": "The time interval since now for which data must be retained (required)." }, "cols": 3, "rows": 1 } [/block] For example: [block:code] { "codes": [ { "code": "local st = require(\"telemetry/storage\")\nlocal series = st.series(\"test\")\n\n-- Keep the last day's worth of data via interval\nseries.trimSince(\"1d\")\n\n-- Keep at most the last 10,000 data points\nseries.trimCount(10000)", "language": "lua" } ] } [/block] [block:callout] { "type": "warning", "title": "Use `trim` functions with care!", "body": "A call to `trimCount()` or `trimSince()` locks the entire Agent database while old rows are purged. The Agent compensates for this by queueing other operations (including the insertion of new data points) until after `trim` finishes executing.\n\nThis should not normally a problem, but under extreme loads, or when `trim` is executed too frequently, it could cause the insertion of data points out of sequence, affecting the results of querying operations until all data points have finished queueing into the database.\n\nTo minimize the chance of inconsistencies in the data, we recommend that you only call `trimCount()` or `trimSince()` when absolutely necessary." } [/block]