{"_id":"57fcc4870312b20e00ac64e9","project":"5435687035740020002a1c04","version":{"_id":"57fcc4860312b20e00ac64c0","project":"5435687035740020002a1c04","__v":1,"createdAt":"2016-10-11T10:52:54.637Z","releaseDate":"2016-10-11T10:52:54.637Z","categories":["57fcc4860312b20e00ac64c1","57fcc4860312b20e00ac64c2","57fcc4860312b20e00ac64c3","57fcc4860312b20e00ac64c4","57fcc4860312b20e00ac64c5","57fcc4860312b20e00ac64c6","57fcc4860312b20e00ac64c7","57fcc4860312b20e00ac64c8","57fcc4860312b20e00ac64c9","57fcc4860312b20e00ac64ca","57fcc4860312b20e00ac64cb"],"is_deprecated":false,"is_hidden":false,"is_beta":false,"is_stable":true,"codename":"[APP-1265], [APP-1035]","version_clean":"5.2.0","version":"5.2"},"category":{"_id":"57fcc4860312b20e00ac64c7","__v":0,"version":"57fcc4860312b20e00ac64c0","project":"5435687035740020002a1c04","sync":{"url":"","isSync":false},"reference":false,"createdAt":"2014-11-25T22:05:12.981Z","from_sync":false,"order":6,"slug":"agent-integration","title":"Data Agent"},"parentDoc":null,"user":"547d27d12eaee50800ed1157","__v":0,"updates":[],"next":{"pages":[],"description":""},"createdAt":"2015-01-29T00:51:12.254Z","link_external":false,"link_url":"","githubsync":"","sync_unique":"","hidden":false,"api":{"results":{"codes":[]},"settings":"","auth":"required","params":[],"url":""},"isReference":false,"order":3,"body":"## Configuration File\n\nThe agent is driven by a simple configuration file written in [TOML Format](https://github.com/toml-lang/toml) and should be kept local to the Agent so it can access the configuration file on startup.\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"#\\n# Server Configuration Parameters\\n#\\n\\n[server]\\n# API token found in the Agents tab in manager.telemetryapp.com\\napi_token = \\\"abc123abc123abc123abc123abc123abc123\\\"\\n# Submit new metrics in a batch at most every 1 second\\nsubmission_interval = \\\"1s\\\"\\n\\n[data]\\n# Location to store\\n# path = \\\"c:\\\\\\\\telemetry_agent.db\\\"\\n# WINDOWS USERS - Uncomment the line above and comment or remove the line below\\npath = \\\"/tmp/telemetry_agent.db\\\"\\n# Expire data after 7 days\\nttl = \\\"7d\\\"\\n\\n#\\n# Graphite Configuration Parameters\\n#\\n\\n[graphite]\\n# Listen on UDP port 2003 on all interfaces for Graphite data\\nlisten_udp = \\\":2003\\\"\\n\\n#\\n# Flow Configurations \\n#\\n\\n#\\n# A specific flow\\n#\\n[[flow]]\\n# The flow tag\\ntag = \\\"graphite_gauge_1\\\"\\n# run the script every 2 seconds\\ninterval = \\\"2s\\\"\\n# The Lua script to run to generate updates\\nscript = \\\"graphite.lua\\\"\",\n      \"language\": \"toml\"\n    }\n  ]\n}\n[/block]\nAbove you'll see an example Agent configuration file used within our [Graphite Integration Example](doc:graphite-example)\n\n# Server Configuration Block\n\nYou begin a server configuration block by specifying [server] in the configuration file.  It has the following parameters.   This block is required.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Parameter\",\n    \"h-1\": \"Required\",\n    \"h-2\": \"Type\",\n    \"0-0\": \"api_token\",\n    \"0-1\": \"Required if you are not running the Agent in listener mode.\",\n    \"0-2\": \"API Token as a string\",\n    \"h-3\": \"Description\",\n    \"0-3\": \"This is the API Token obtained by creating a new Agent in the manager\",\n    \"1-0\": \"submission_interval\",\n    \"1-1\": \"\",\n    \"1-2\": \"Time string (\\\"1s\\\", \\\"5h\\\", \\\"7d\\\", \\\"3w\\\")\",\n    \"1-3\": \"The minimum interval to batch metric submissions on.  The agent will not submit metrics faster than this interval.\"\n  },\n  \"cols\": 4,\n  \"rows\": 2\n}\n[/block]\nYou can create a new Agent and obtain an API token by visiting app.telemetrytv.com and adding a new Token within **Settings > [Account Name] > API Tokens** .\n[block:image]\n{\n  \"images\": [\n    {\n      \"image\": [\n        \"https://files.readme.io/083b010-api_tokens.jpg\",\n        \"api_tokens.jpg\",\n        2660,\n        1200,\n        \"#202830\"\n      ]\n    }\n  ]\n}\n[/block]\n\n[block:callout]\n{\n  \"type\": \"info\",\n  \"title\": \"API Token Environment Variable\",\n  \"body\": \"For security reasons you may optionally provide the API Token in the environment variables by setting TELEMETRY_API_TOKEN in the environment before running the script.   The configuration file takes precedence if the token is set in both places.\"\n}\n[/block]\nRather than starting a new **HTTPS** connection to the API every time a job wishes to submit an update, the Agent automatically coalesces all updates with a specific period of time into a single API call, which reduces your API usage and limits the load on the API itself.\n\nGenerally speaking, longer submission intervals will result in less API usage, at the cost of higher data latency. Lower submission intervals, on the other hand, will ensure that data is sent to your boards more quickly after it is generated, but that more API calls will be made.\n[block:callout]\n{\n  \"type\": \"info\",\n  \"body\": \"The default submission interval of one second is well below the usage limits imposed by our API for concurrent requests—thus allowing you plenty of margin to run multiple agents, even if their requests are routed through a single IP.\\n\\nShould you run up against rate limiting, the API will signal the Agent by returning a 429 HTTP code; under this circumstance, the Agent will produce a warning and automatically implement a graceful back-off algorithm to reduce strain on resources.\",\n  \"title\": \"API Limits\"\n}\n[/block]\n# API Listener Block\n\nThe Agent can be optionally controlled remotely using the TelemetryTV app within the [Agent Interface](doc:user-interface). To do this you must be running the Agent API. The API will start automatically when the Agent is executed without a configuration file or API token. If an API token has been specified then the following configuration block must be present:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Parameter\",\n    \"h-1\": \"Required\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Description\",\n    \"0-0\": \"auth_key\",\n    \"0-1\": \"Required if an `api_token` has been specified and you want to run the Agent in listener mode.\",\n    \"0-2\": \"string\",\n    \"0-3\": \"A 16+ character string to be used as a passphrase for accessing the Agent's internal API. If not supplied and no `api_token` is present then a 16 character `auth_token` will be randomly generated.\",\n    \"1-0\": \"listen\",\n    \"1-2\": \"string\",\n    \"1-3\": \"The port number that the Agent API will use to listen for incoming connections. In the format of \\\":####\\\". If not provided then `listen` will default to \\\":9800\\\".\"\n  },\n  \"cols\": 4,\n  \"rows\": 2\n}\n[/block]\n# Data Configuration Block \n\nThe agent can optionally cache data on your local filesystem.  This is typically used by the agent to store Graphite metrics for aggregation, anomaly detection and the like.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Parameter\",\n    \"h-1\": \"Required\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Description\",\n    \"0-0\": \"path\",\n    \"1-0\": \"ttl\",\n    \"0-2\": \"Filsystem path string\",\n    \"0-3\": \"The filesystem path for where to store the database.\",\n    \"1-2\": \"Time string (\\\"1s\\\", \\\"5h\\\", \\\"7d\\\", \\\"3w\\\")\",\n    \"1-3\": \"How long to keep data for.  Data older than this period will be removed from the database.\"\n  },\n  \"cols\": 4,\n  \"rows\": 2\n}\n[/block]\nThe path you provide here should point to the name of a file—its extension doesn't matter. The Agent will create a few additional files with the same name but a different extension in the same directory. Thus, the user account under which the Agent runs should have write and create permissions on the directory.\n\nMake sure that the filesystem on which the data path resides supports file locking.\n\nThe **ttl** parameter defines the default age after which the Agent automatically discards metrics in order to prevent the database from growing indefinitely. Since it affects _every_ data point in the system, you should set it to the highest possible value.\n\n# Graphite Configuration Block \n\nThis optional configuration block specifies parameters for the Graphite support for the agent.\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Parameter\",\n    \"h-1\": \"Required\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Description\",\n    \"0-0\": \"listen_udp\",\n    \"1-0\": \"listen_tcp\",\n    \"0-2\": \"hostname:port string combination\",\n    \"1-2\": \"hostname:port string combination\",\n    \"0-3\": \"Sets the UDP port to listen on for incoming Graphite data\",\n    \"1-3\": \"Sets the TCP port to listen on for incoming Graphite data\"\n  },\n  \"cols\": 4,\n  \"rows\": 2\n}\n[/block]\nThe Agent expects Graphite metrics to be sent to it over the network; this approach works well both in the case in which the Agent runs on the same machine as the process that generates the metrics, and in the case where the Agent runs on dedicated hardware and must collect information from multiple sources distributed across multiple servers.\n\nMetrics can be sent both over UDP or TCP using a simple plain-text protocol that is compatible with the [Graphite](http://graphite.wikidot.com) open-source tool; thus, the Agent can function as a drop-in replacement for a Graphite server in your existing infrastructure.\n\n# Flow Configuration Block \n\nThe flow configuration block specifies one or more flows in the format of [[flow]]  (note the double square brackets, this specifies multiple items in an array in the TOML language).  Each flow must have either a script or exec definition which provides metrics for the flow.  \n\nFlows obtain metrics in one of two ways:  either by executing an external program or by running a Lua script (which in turn gathers data from outside sources).   \n\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Parameter\",\n    \"h-1\": \"Required\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Description\",\n    \"0-0\": \"tag\",\n    \"0-1\": \"Required if passing data to a single flow\",\n    \"0-2\": \"Flow Tag as String\",\n    \"3-2\": \"Time string (\\\"1s\\\", \\\"5h\\\", \\\"7d\\\", \\\"3w\\\")\",\n    \"3-0\": \"interval\",\n    \"5-0\": \"script\",\n    \"6-0\": \"exec\",\n    \"5-2\": \"Filesystem path string to Lua script\",\n    \"6-2\": \"Filesystem path string to executable\",\n    \"0-3\": \"This is the flow tag,  you can find the flow tag by clicking on a widget in the manager application.\",\n    \"3-3\": \"How frequently to update the flow.  You'll need to set this equal or greater than the submission_interval to have effect.\",\n    \"5-3\": \"The Lua script to execute in order to obtain metrics for this flow.\",\n    \"6-3\": \"An external executable like a node.js, ruby or python script to run in order to obtain metrics for this flow.  The script must return a flow data structure in standard out.\",\n    \"7-0\": \"expiration\",\n    \"7-2\": \"Integer\",\n    \"7-3\": \"Seconds from when the flow is submitted to mark it expired.\",\n    \"11-0\": \"[flow.args]\",\n    \"11-2\": \"dictionary\",\n    \"11-3\": \"Optionally specify arguments to the Lua script.\",\n    \"8-0\": \"variant\",\n    \"8-2\": \"string\",\n    \"8-3\": \"Optionally ensure the targeted flow is of a specific variant.\",\n    \"10-0\": \"[flow.template]\",\n    \"10-2\": \"dictionary\",\n    \"10-3\": \"Optionally specify a template to create a flow from if it does not exist.\",\n    \"4-0\": \"channel_tag\",\n    \"4-2\": \"Channel ID as String\",\n    \"4-3\": \"This parameter will force the flow to only update for the channel associated with the supplied channel iD.\",\n    \"1-0\": \"id\",\n    \"1-1\": \"Required if sending a batch payload or if running a flow-less job\",\n    \"1-2\": \"String\",\n    \"1-3\": \"This acts as the unique identifier for your job when no Flow Tag is present.\",\n    \"2-0\": \"batch\",\n    \"2-2\": \"Boolean\",\n    \"2-3\": \"Set to `true` if your script or executable will output a batch flow payload. Otherwise this property can be omitted.\",\n    \"9-0\": \"args\",\n    \"9-2\": \"array\",\n    \"9-3\": \"Optionally specify an array of command line arguments to pass to an executable.\"\n  },\n  \"cols\": 4,\n  \"rows\": 12\n}\n[/block]\nChanges to Lua scripts are detected automatically and the script is reloaded on the next interval,  you do not need to restart the agent.\n\nIt is important to note that the interval is deterministic in nature, that is, successive iterations of a job are reasonably guaranteed to take place at exactly the interval you specify. If the job's external process is still running by the time the next iteration is scheduled for execution, the Agent will skip it altogether, output a warning, and wait for the next scheduled execution time.\n\n# oAuth Configuration Block\n\nThird party APIs that require [oAuth authentication](doc:oauth) must have an entry in the Agent configuration file. These entries are TOML blocks in the format of [oauth.entry_name]. The `entry_name` is the name that will be used to store the entry in the Agent's database and [perform GET and POST requests](doc:accessing-third-party-apis-from-lua) within Lua. The block can contain the following properties:\n[block:parameters]\n{\n  \"data\": {\n    \"h-0\": \"Parameter\",\n    \"h-1\": \"Required\",\n    \"h-2\": \"Type\",\n    \"h-3\": \"Description\",\n    \"0-2\": \"Integer\",\n    \"0-0\": \"version\",\n    \"1-0\": \"client_id\",\n    \"2-0\": \"client_secret\",\n    \"3-0\": \"authorization_url\",\n    \"4-0\": \"token_url\",\n    \"6-0\": \"scopes\",\n    \"8-0\": \"header\",\n    \"6-2\": \"Array of strings\",\n    \"1-2\": \"String\",\n    \"2-2\": \"String\",\n    \"3-2\": \"String\",\n    \"4-2\": \"String\",\n    \"8-2\": \"Dictionary\",\n    \"0-1\": \"Required\",\n    \"1-1\": \"Required\",\n    \"2-1\": \"Required\",\n    \"3-1\": \"Required\",\n    \"4-1\": \"Required\",\n    \"1-3\": \"Identifier for the application that you are connecting with the Agent. The name of the parameter varies depending on the service. May be called Consumer Key, App ID, etc...\",\n    \"0-3\": \"This field can be one of two values and specifies if the API is using oAuth version `1` or `2`.\",\n    \"2-3\": \"The secure key that acts as a password for the client_id.\",\n    \"5-0\": \"ttl\",\n    \"5-1\": \"\",\n    \"3-3\": \"Endpoint to authenticate your oAuth credentials. Provided in the third party services documentation.\",\n    \"4-3\": \"Endpoint for retrieving your authentication token. Provided in the third party services documentation.\",\n    \"5-3\": \"Used to produce an expiration timestamp for the oAuth token. Time is relative to moment from when authentication takes place.\",\n    \"5-2\": \"Time string (\\\"1s\\\", \\\"5h\\\", \\\"7d\\\", \\\"3w\\\")\",\n    \"6-3\": \"Array of access scope URIs.\",\n    \"8-3\": \"Key/value strings of HTTP header values to be submitted with oAuth v1 requests.\",\n    \"9-3\": \"Specifies the oAuth v1 signature method. Can be one of `hmac_sha1`, `rsa_sha1`, or `plaintext`.\",\n    \"9-2\": \"String\",\n    \"9-0\": \"signature_method\",\n    \"7-0\": \"credentials_url\",\n    \"7-1\": \"Required for oAuth v1. Not used in v2\",\n    \"7-2\": \"String\",\n    \"7-3\": \"oAuth v1 requires an additional request token step. This endpoint is used in that process. Provided in the third party API's documentation.\"\n  },\n  \"cols\": 4,\n  \"rows\": 10\n}\n[/block]\n## Passing command-line argument to external processes\n\nBy default, the Agent invokes the external process associated with a job without any argument. You can, however, specify arguments in the configuration file. For example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"...\\n[[flow]]\\ntag = \\\"testing\\\"\\nexec = \\\"myscript.js\\\"\\nargs = [\\\"arg1\\\", \\\"arg2\\\", \\\"arg3\\\"]\",\n      \"language\": \"toml\"\n    }\n  ]\n}\n[/block]\n\n## Making the Agent Create and Validate Flows\n\nBy default, the Agent assumes that the flow whose tag you specify exists, and that the external process outputs data that will be compatible with its variant and contents. If the flow does not exist, or if its variant doesn't match the external process' output, API updates will fail and errors will be printed in the log.\n\nAs an alternative, you can request that the Agent verify that the flow exists and is of an expected variant type at startup:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"[[flow]]\\ntag = \\\"testing\\\"\\nexec = \\\"myscript.js\\\"\\nvariant = \\\"number\\\"\",\n      \"language\": \"toml\"\n    }\n  ]\n}\n[/block]\nIf the flow exists and it is not of the correct variant, the Agent will print an error to the log and prevent the job from being scheduled.  If the flow doesn't exist then the agent will create the flow for you.\n\n## Auto Create Flows with Templates\n\nYou can also provide a template that the Agent can use to create the flow for you if it doesn't exist:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"...\\n[[flow]]\\ntag = \\\"testing\\\"\\nexec = \\\"myscript.js\\\"\\nvariant = \\\"number\\\"\\n[flow.template]\\n\\tlabel.text = \\\"My Number\\\"\",\n      \"language\": \"toml\"\n    }\n  ]\n}\n[/block]\nIn this case, if the Agent can't find the flow, it will create it for you. (Note, however, that if it does exist and is the wrong variant, the Agent will still print an error and refuse to schedule the flow)\n\n## Flow-less Jobs\n\nAgents can run jobs that process data but are not expected to return a flow.\n\n## Batch Flows\n\nSometimes it is preferable to generate flow data for multiple jobs within the same script or executable. Batch flows allow for you to attach several payloads to a single JSON object and update all attached flows at the same time. For example:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"[[flow]]\\nid = \\\"A test batch flow\\\"\\nbatch = true\\ninterval = \\\"2s\\\"\\nscript = \\\"batchflow.lua\\\"\\n[flow.args]\\n  flow_tag_1 = \\\"batch_test_text_1\\\"\\n  flow_tag_2 = \\\"batch_test_gauge_1\\\"\",\n      \"language\": \"toml\"\n    }\n  ]\n}\n[/block]\nThis could be used to parse the following Lua script:\n[block:code]\n{\n  \"codes\": [\n    {\n      \"code\": \"output = {\\n  [args.flow_tag_1] = {text = \\\"Batch text\\\"},\\n  [args.flow_tag_2] = {number = {value = (math.random() + math.random(1, 100))}}\\n}\",\n      \"language\": \"lua\",\n      \"name\": null\n    }\n  ]\n}\n[/block]\nNote that in this example we are passing `flow_tag_1` and `flow_tag_2` as arguments. Hard coding Flow Tags within scripts and executables should be avoided. Flow Tags should always be set at the configuration level so that they can be easily located and modified.\n\n# Running the Agent \n\nThe Agent is fairly straightforward​ to run, you just need to execute the binary with the configuration file specified as such:\n\n    gotelemetry_agent run -c config.toml","excerpt":"","slug":"agent-configuration-file","type":"basic","title":"Configuration"}
## Configuration File The agent is driven by a simple configuration file written in [TOML Format](https://github.com/toml-lang/toml) and should be kept local to the Agent so it can access the configuration file on startup. [block:code] { "codes": [ { "code": "#\n# Server Configuration Parameters\n#\n\n[server]\n# API token found in the Agents tab in manager.telemetryapp.com\napi_token = \"abc123abc123abc123abc123abc123abc123\"\n# Submit new metrics in a batch at most every 1 second\nsubmission_interval = \"1s\"\n\n[data]\n# Location to store\n# path = \"c:\\\\telemetry_agent.db\"\n# WINDOWS USERS - Uncomment the line above and comment or remove the line below\npath = \"/tmp/telemetry_agent.db\"\n# Expire data after 7 days\nttl = \"7d\"\n\n#\n# Graphite Configuration Parameters\n#\n\n[graphite]\n# Listen on UDP port 2003 on all interfaces for Graphite data\nlisten_udp = \":2003\"\n\n#\n# Flow Configurations \n#\n\n#\n# A specific flow\n#\n[[flow]]\n# The flow tag\ntag = \"graphite_gauge_1\"\n# run the script every 2 seconds\ninterval = \"2s\"\n# The Lua script to run to generate updates\nscript = \"graphite.lua\"", "language": "toml" } ] } [/block] Above you'll see an example Agent configuration file used within our [Graphite Integration Example](doc:graphite-example) # Server Configuration Block You begin a server configuration block by specifying [server] in the configuration file. It has the following parameters. This block is required. [block:parameters] { "data": { "h-0": "Parameter", "h-1": "Required", "h-2": "Type", "0-0": "api_token", "0-1": "Required if you are not running the Agent in listener mode.", "0-2": "API Token as a string", "h-3": "Description", "0-3": "This is the API Token obtained by creating a new Agent in the manager", "1-0": "submission_interval", "1-1": "", "1-2": "Time string (\"1s\", \"5h\", \"7d\", \"3w\")", "1-3": "The minimum interval to batch metric submissions on. The agent will not submit metrics faster than this interval." }, "cols": 4, "rows": 2 } [/block] You can create a new Agent and obtain an API token by visiting app.telemetrytv.com and adding a new Token within **Settings > [Account Name] > API Tokens** . [block:image] { "images": [ { "image": [ "https://files.readme.io/083b010-api_tokens.jpg", "api_tokens.jpg", 2660, 1200, "#202830" ] } ] } [/block] [block:callout] { "type": "info", "title": "API Token Environment Variable", "body": "For security reasons you may optionally provide the API Token in the environment variables by setting TELEMETRY_API_TOKEN in the environment before running the script. The configuration file takes precedence if the token is set in both places." } [/block] Rather than starting a new **HTTPS** connection to the API every time a job wishes to submit an update, the Agent automatically coalesces all updates with a specific period of time into a single API call, which reduces your API usage and limits the load on the API itself. Generally speaking, longer submission intervals will result in less API usage, at the cost of higher data latency. Lower submission intervals, on the other hand, will ensure that data is sent to your boards more quickly after it is generated, but that more API calls will be made. [block:callout] { "type": "info", "body": "The default submission interval of one second is well below the usage limits imposed by our API for concurrent requests—thus allowing you plenty of margin to run multiple agents, even if their requests are routed through a single IP.\n\nShould you run up against rate limiting, the API will signal the Agent by returning a 429 HTTP code; under this circumstance, the Agent will produce a warning and automatically implement a graceful back-off algorithm to reduce strain on resources.", "title": "API Limits" } [/block] # API Listener Block The Agent can be optionally controlled remotely using the TelemetryTV app within the [Agent Interface](doc:user-interface). To do this you must be running the Agent API. The API will start automatically when the Agent is executed without a configuration file or API token. If an API token has been specified then the following configuration block must be present: [block:parameters] { "data": { "h-0": "Parameter", "h-1": "Required", "h-2": "Type", "h-3": "Description", "0-0": "auth_key", "0-1": "Required if an `api_token` has been specified and you want to run the Agent in listener mode.", "0-2": "string", "0-3": "A 16+ character string to be used as a passphrase for accessing the Agent's internal API. If not supplied and no `api_token` is present then a 16 character `auth_token` will be randomly generated.", "1-0": "listen", "1-2": "string", "1-3": "The port number that the Agent API will use to listen for incoming connections. In the format of \":####\". If not provided then `listen` will default to \":9800\"." }, "cols": 4, "rows": 2 } [/block] # Data Configuration Block The agent can optionally cache data on your local filesystem. This is typically used by the agent to store Graphite metrics for aggregation, anomaly detection and the like. [block:parameters] { "data": { "h-0": "Parameter", "h-1": "Required", "h-2": "Type", "h-3": "Description", "0-0": "path", "1-0": "ttl", "0-2": "Filsystem path string", "0-3": "The filesystem path for where to store the database.", "1-2": "Time string (\"1s\", \"5h\", \"7d\", \"3w\")", "1-3": "How long to keep data for. Data older than this period will be removed from the database." }, "cols": 4, "rows": 2 } [/block] The path you provide here should point to the name of a file—its extension doesn't matter. The Agent will create a few additional files with the same name but a different extension in the same directory. Thus, the user account under which the Agent runs should have write and create permissions on the directory. Make sure that the filesystem on which the data path resides supports file locking. The **ttl** parameter defines the default age after which the Agent automatically discards metrics in order to prevent the database from growing indefinitely. Since it affects _every_ data point in the system, you should set it to the highest possible value. # Graphite Configuration Block This optional configuration block specifies parameters for the Graphite support for the agent. [block:parameters] { "data": { "h-0": "Parameter", "h-1": "Required", "h-2": "Type", "h-3": "Description", "0-0": "listen_udp", "1-0": "listen_tcp", "0-2": "hostname:port string combination", "1-2": "hostname:port string combination", "0-3": "Sets the UDP port to listen on for incoming Graphite data", "1-3": "Sets the TCP port to listen on for incoming Graphite data" }, "cols": 4, "rows": 2 } [/block] The Agent expects Graphite metrics to be sent to it over the network; this approach works well both in the case in which the Agent runs on the same machine as the process that generates the metrics, and in the case where the Agent runs on dedicated hardware and must collect information from multiple sources distributed across multiple servers. Metrics can be sent both over UDP or TCP using a simple plain-text protocol that is compatible with the [Graphite](http://graphite.wikidot.com) open-source tool; thus, the Agent can function as a drop-in replacement for a Graphite server in your existing infrastructure. # Flow Configuration Block The flow configuration block specifies one or more flows in the format of [[flow]] (note the double square brackets, this specifies multiple items in an array in the TOML language). Each flow must have either a script or exec definition which provides metrics for the flow. Flows obtain metrics in one of two ways: either by executing an external program or by running a Lua script (which in turn gathers data from outside sources). [block:parameters] { "data": { "h-0": "Parameter", "h-1": "Required", "h-2": "Type", "h-3": "Description", "0-0": "tag", "0-1": "Required if passing data to a single flow", "0-2": "Flow Tag as String", "3-2": "Time string (\"1s\", \"5h\", \"7d\", \"3w\")", "3-0": "interval", "5-0": "script", "6-0": "exec", "5-2": "Filesystem path string to Lua script", "6-2": "Filesystem path string to executable", "0-3": "This is the flow tag, you can find the flow tag by clicking on a widget in the manager application.", "3-3": "How frequently to update the flow. You'll need to set this equal or greater than the submission_interval to have effect.", "5-3": "The Lua script to execute in order to obtain metrics for this flow.", "6-3": "An external executable like a node.js, ruby or python script to run in order to obtain metrics for this flow. The script must return a flow data structure in standard out.", "7-0": "expiration", "7-2": "Integer", "7-3": "Seconds from when the flow is submitted to mark it expired.", "11-0": "[flow.args]", "11-2": "dictionary", "11-3": "Optionally specify arguments to the Lua script.", "8-0": "variant", "8-2": "string", "8-3": "Optionally ensure the targeted flow is of a specific variant.", "10-0": "[flow.template]", "10-2": "dictionary", "10-3": "Optionally specify a template to create a flow from if it does not exist.", "4-0": "channel_tag", "4-2": "Channel ID as String", "4-3": "This parameter will force the flow to only update for the channel associated with the supplied channel iD.", "1-0": "id", "1-1": "Required if sending a batch payload or if running a flow-less job", "1-2": "String", "1-3": "This acts as the unique identifier for your job when no Flow Tag is present.", "2-0": "batch", "2-2": "Boolean", "2-3": "Set to `true` if your script or executable will output a batch flow payload. Otherwise this property can be omitted.", "9-0": "args", "9-2": "array", "9-3": "Optionally specify an array of command line arguments to pass to an executable." }, "cols": 4, "rows": 12 } [/block] Changes to Lua scripts are detected automatically and the script is reloaded on the next interval, you do not need to restart the agent. It is important to note that the interval is deterministic in nature, that is, successive iterations of a job are reasonably guaranteed to take place at exactly the interval you specify. If the job's external process is still running by the time the next iteration is scheduled for execution, the Agent will skip it altogether, output a warning, and wait for the next scheduled execution time. # oAuth Configuration Block Third party APIs that require [oAuth authentication](doc:oauth) must have an entry in the Agent configuration file. These entries are TOML blocks in the format of [oauth.entry_name]. The `entry_name` is the name that will be used to store the entry in the Agent's database and [perform GET and POST requests](doc:accessing-third-party-apis-from-lua) within Lua. The block can contain the following properties: [block:parameters] { "data": { "h-0": "Parameter", "h-1": "Required", "h-2": "Type", "h-3": "Description", "0-2": "Integer", "0-0": "version", "1-0": "client_id", "2-0": "client_secret", "3-0": "authorization_url", "4-0": "token_url", "6-0": "scopes", "8-0": "header", "6-2": "Array of strings", "1-2": "String", "2-2": "String", "3-2": "String", "4-2": "String", "8-2": "Dictionary", "0-1": "Required", "1-1": "Required", "2-1": "Required", "3-1": "Required", "4-1": "Required", "1-3": "Identifier for the application that you are connecting with the Agent. The name of the parameter varies depending on the service. May be called Consumer Key, App ID, etc...", "0-3": "This field can be one of two values and specifies if the API is using oAuth version `1` or `2`.", "2-3": "The secure key that acts as a password for the client_id.", "5-0": "ttl", "5-1": "", "3-3": "Endpoint to authenticate your oAuth credentials. Provided in the third party services documentation.", "4-3": "Endpoint for retrieving your authentication token. Provided in the third party services documentation.", "5-3": "Used to produce an expiration timestamp for the oAuth token. Time is relative to moment from when authentication takes place.", "5-2": "Time string (\"1s\", \"5h\", \"7d\", \"3w\")", "6-3": "Array of access scope URIs.", "8-3": "Key/value strings of HTTP header values to be submitted with oAuth v1 requests.", "9-3": "Specifies the oAuth v1 signature method. Can be one of `hmac_sha1`, `rsa_sha1`, or `plaintext`.", "9-2": "String", "9-0": "signature_method", "7-0": "credentials_url", "7-1": "Required for oAuth v1. Not used in v2", "7-2": "String", "7-3": "oAuth v1 requires an additional request token step. This endpoint is used in that process. Provided in the third party API's documentation." }, "cols": 4, "rows": 10 } [/block] ## Passing command-line argument to external processes By default, the Agent invokes the external process associated with a job without any argument. You can, however, specify arguments in the configuration file. For example: [block:code] { "codes": [ { "code": "...\n[[flow]]\ntag = \"testing\"\nexec = \"myscript.js\"\nargs = [\"arg1\", \"arg2\", \"arg3\"]", "language": "toml" } ] } [/block] ## Making the Agent Create and Validate Flows By default, the Agent assumes that the flow whose tag you specify exists, and that the external process outputs data that will be compatible with its variant and contents. If the flow does not exist, or if its variant doesn't match the external process' output, API updates will fail and errors will be printed in the log. As an alternative, you can request that the Agent verify that the flow exists and is of an expected variant type at startup: [block:code] { "codes": [ { "code": "[[flow]]\ntag = \"testing\"\nexec = \"myscript.js\"\nvariant = \"number\"", "language": "toml" } ] } [/block] If the flow exists and it is not of the correct variant, the Agent will print an error to the log and prevent the job from being scheduled. If the flow doesn't exist then the agent will create the flow for you. ## Auto Create Flows with Templates You can also provide a template that the Agent can use to create the flow for you if it doesn't exist: [block:code] { "codes": [ { "code": "...\n[[flow]]\ntag = \"testing\"\nexec = \"myscript.js\"\nvariant = \"number\"\n[flow.template]\n\tlabel.text = \"My Number\"", "language": "toml" } ] } [/block] In this case, if the Agent can't find the flow, it will create it for you. (Note, however, that if it does exist and is the wrong variant, the Agent will still print an error and refuse to schedule the flow) ## Flow-less Jobs Agents can run jobs that process data but are not expected to return a flow. ## Batch Flows Sometimes it is preferable to generate flow data for multiple jobs within the same script or executable. Batch flows allow for you to attach several payloads to a single JSON object and update all attached flows at the same time. For example: [block:code] { "codes": [ { "code": "[[flow]]\nid = \"A test batch flow\"\nbatch = true\ninterval = \"2s\"\nscript = \"batchflow.lua\"\n[flow.args]\n flow_tag_1 = \"batch_test_text_1\"\n flow_tag_2 = \"batch_test_gauge_1\"", "language": "toml" } ] } [/block] This could be used to parse the following Lua script: [block:code] { "codes": [ { "code": "output = {\n [args.flow_tag_1] = {text = \"Batch text\"},\n [args.flow_tag_2] = {number = {value = (math.random() + math.random(1, 100))}}\n}", "language": "lua", "name": null } ] } [/block] Note that in this example we are passing `flow_tag_1` and `flow_tag_2` as arguments. Hard coding Flow Tags within scripts and executables should be avoided. Flow Tags should always be set at the configuration level so that they can be easily located and modified. # Running the Agent The Agent is fairly straightforward​ to run, you just need to execute the binary with the configuration file specified as such: gotelemetry_agent run -c config.toml