{"id":2257,"date":"2013-07-13T18:29:40","date_gmt":"2013-07-14T01:29:40","guid":{"rendered":"https:\/\/mathpirate.net\/log\/?p=2257"},"modified":"2013-07-13T18:29:40","modified_gmt":"2013-07-14T01:29:40","slug":"overwriting-anomalous-data-in-graphite","status":"publish","type":"post","link":"https:\/\/mathpirate.net\/log\/2013\/07\/13\/overwriting-anomalous-data-in-graphite\/","title":{"rendered":"Overwriting Anomalous Data in Graphite"},"content":{"rendered":"

Recently, I’ve been setting up a home sensor array using Arduino modules, which feed data into a Graphite instance running on a Raspberry Pi.\u00c2\u00a0 Along the way, through faulty wiring, faulty coding, or the effects of sunspots, I’ve ended up with some bad data ending up in Graphite.<\/p>\n

And by bad data, I mean that my living room sensor once reported that it was 289,268 degrees C one day.\u00c2\u00a0 Since my house and the surrounding neighborhood show no signs of having been\u00c2\u00a0vaporized, I am forced to believe that reading is incorrect.<\/p>\n

\"\"<\/a><\/p>\n

Since having a temperature reading of several hundred thousand degrees throws off the range of the graph somewhat, I wanted to get rid of that errant data point. \u00c2\u00a0I looked around for a Whisper DB editor, but didn’t find anything. \u00c2\u00a0Graphite’s all open source, so I probably could build one myself, but I really didn’t feel like going that far. \u00c2\u00a0Then I remembered the “Feeding in Your Data” page in the Graphite docs, where it talks about using the command line to send data values. \u00c2\u00a0Maybe if you can write datapoints using this method, you can overwrite datapoints using this method, too. \u00c2\u00a0So, I figured I’d give it a shot.<\/p>\n

The example they gave on the page is this:<\/p>\n

echo \"local.random.diceroll 4 `date +%s`\" | nc ${SERVER} ${PORT};<\/pre>\n
The “local.random.diceroll” part is your counter name.<\/p>\n
The “4” part is the data you want to write.<\/p>\n
And the “date +%s” is just a fancy way of saying the Unix Epoch Timestamp of Now.<\/p>\n
So, in my case, I’d overwrite the counter “stats.gauges.Temperature.LivingRoom” with a comfortable room temparature value of somewhere between 20 and 25. \u00c2\u00a0But what about the timestamp? \u00c2\u00a0I can’t just go in and hope I get the timestamp. \u00c2\u00a0I can guess from that graph that it happened sometime around 00:32:30, but who knows if I’m right. \u00c2\u00a0And what timezone is Whisper using? \u00c2\u00a0Local time? \u00c2\u00a0UTC? \u00c2\u00a0And if it’s UTC, is that + 8 hours or +7 hours this time of year…? \u00c2\u00a0I needed a better way to get the exact timestamp.<\/p>\n
Fortunately, Graphite makes that relatively easy. \u00c2\u00a0You see, you can get the data in json format if you want. \u00c2\u00a0While it’s not as pretty as the graph, it does have the exact values of the timestamp you’re looking for. \u00c2\u00a0Getting the json is fairly straightforward. \u00c2\u00a0First, you go to the Graphite Browser and find the data you want to fix. \u00c2\u00a0Once you have it in the window, right click and open the graph in a new tab (Or copy\/paste the graph URL, either way works). \u00c2\u00a0From there, edit the URL of the graph, by adding “&format=json” to the end. \u00c2\u00a0Load that page and your screen will fill with json datapoints. \u00c2\u00a0Like so:<\/p>\n
\"\"<\/a><\/p>\n
In that sea of numbers, find the value you want to replace. \u00c2\u00a0In my case, it’s easy to find, because it’s the one claiming that my living room jumped up to 6 times hotter than the surface of the sun for a moment. \u00c2\u00a0The second value in that datapoint is the timestamp I need to wipe out the errant temperature spike. \u00c2\u00a0Now that I have it, I can run the following command to put things right:<\/p>\n
echo \"stats.gauges.Temperature.LivingRoom 22.8 1373614500\" | nc localhost 2003<\/pre>\n
And now my living room is back to a comfortable 22.8 degrees.<\/p>\n
\"\"<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Recently, I’ve been setting up a home sensor array using Arduino modules, which feed data into a Graphite instance running on a Raspberry Pi.\u00c2\u00a0 Along the way, through faulty wiring, faulty coding, or the effects of sunspots, I’ve ended up with some bad data ending up in Graphite. And by bad data, I mean that […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[7,1],"tags":[218,217],"_links":{"self":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts\/2257"}],"collection":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/comments?post=2257"}],"version-history":[{"count":3,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts\/2257\/revisions"}],"predecessor-version":[{"id":2265,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/posts\/2257\/revisions\/2265"}],"wp:attachment":[{"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/media?parent=2257"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/categories?post=2257"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mathpirate.net\/log\/wp-json\/wp\/v2\/tags?post=2257"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}