Tibber Pulse Data Format

[johu]

Since May 15th we get our electricity from Tibber. To participate from (low) spot market prices they have developed a gadget that reads the IR data stream from the energy meter (comparable to this: viewtopic.php?t=2322). The data is then sent to the so called bridge, which receives the data via some 433 MHz protocol and then sends it to Tibbers MQTT server.

Trouble is, my energy meter is not supported by Pulse so right now I can't use the hourly spot market prices (as we speak 4 ct/kWh because lots of wind and solar meets a bank holiday).

So I'd like to talk to Tibbers MQTT server directly and send the data I read anyway.

I have successfully logged in to their server but I don't know what data format they expect in the message. Does anyone here use tibber pulse and feels like intercepting the message? (The equivalent of a CAN log, a few frames annotated with the real data)

More into on how to do that here: https://blog.wyraz.de/allgemein/a-brief ... se-bridge/ and here https://github.com/MSkjel/LocalPulse2Tibber

[Zapatero]

I can support, if you tell me how! Contact me - you know how to find me

[johu]

Thanks

Just one thing I found out the hard way: when I deleted the various keys from the bridge in order to connect to my own MQTT server I found I couldn't add them back in because of some memory overflow.

[johu]

So, I got a bit further with this, be aware that this may apply only to the german Tibber-IR infrared reader.

With the help of others I got the Tibber bridge to connect to my local MQTT server. Besides setting address and port of your local broker you must also fully delete the various crypto keys which is only possible via the console
You do "param_list" which gives you the respective ids, in my case 5, 6 and 7

Then you type "param set 5 \" (same with 6 and 7). That clears the keys and will connect to your broker without cryptography.
Finally you save this with "param_store"

Ok, so now I'm getting some meta data messages on two topics:
$aws/rules/ingest_tibber_bridge_data/tibber-bridge/yyy/publish/TJH01/3494545e84000/event (yyy is your personal ID or something, 3494 is another ID of the tibber bridge itself). The string $aws/rules/ingest_tibber_bridge_data/tibber-bridge/yyy/publish is contained in parameter mqtt_topic
Here's an example:

Code: Select all

{
  "$type" : "event",
  "id" : 10009,
  "class" : "efr32_node_event",
  "name" : "NODE_EVENT_NO_DATA",
  "metrics" : {
    "$type" : "metrics",
    "ssid" : "tach",
    "rssi" : -70,
    "bssid" : "3481c4e31c3f",
    "mqttcon" : 0,
    "wificon" : 0,
    "id" : "3494545e8400",
    "uptime" : 596,
    "efr_uptime" : 12983,
    "vcc" : 3.4779999256134033,
    "vacrms" : 226.4521942138672,
    "heap" : 104692,
    "coredump_available" : true,
    "model" : "TJH01",
    "esp_version" : "1268-f95e5239",
    "efr_version" : "557-dee09095",
    "nodes" : [ {
      "$type" : "node",
      "id" : 1,
      "eui" : "dc8e95fffeb82bfe",
      "model" : "TFD01",
      "version" : "986-49df5891",
      "rssi" : -46.511810302734375,
      "seen" : 9,
      "available" : true,
      "pubcnt" : 0,
      "ota_state" : "idle",
      "ota_status" : "up2date",
      "manifest_version" : "986-49df5891",
      "up2date" : true
    } ],
    "ota" : [ {
      "model" : "tibber-pulse-ir-hub-esp32",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-css",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-js",
      "manifest_version" : "1268-f95e5239",
      "current_version" : "1268-f95e5239",
      "state" : "up2date"
    }, {
      "model" : "tibber-pulse-ir-hub-efr32-fg1",
      "manifest_version" : "557-dee09095",
      "current_version" : "557-dee09095",
      "state" : "up2date"
    } ],
    "hub_ota_status" : "up2date",
    "manifest_state" : "synced",
    "netflags" : 25,
    "ip" : "192.168.188.31",
    "gateway" : "192.168.188.1",
    "dns0" : "192.168.188.1",
    "dns1" : "0.0.0.0",
    "dns2" : "0.0.0.0",
    "mac" : "34:94:54:5e:84:0",
    "ap_mac" : "34:94:54:5e:84:1"
  }
}

There are some other node events especially at bootup and when the IR reader sends something.
Then there is $aws/rules/ingest_tibber_bridge_data/tibber-bridge/yyy/publish/TJH01/3494545e84000/metric:

Code: Select all

{
  "$type" : "node_status",
  "node_status" : {
    "product_id" : 49344,
    "bootloader_version" : 17563650,
    "node_battery_voltage" : 3.458252,
    "node_temperature" : 28.498924,
    "node_avg_rssi" : -47.639168,
    "node_avg_lqi" : 198.696732,
    "acmp_rx_autolevel_300" : 146,
    "acmp_rx_autolevel_9600" : 146,
    "radio_tx_power" : 200,
    "node_uptime_ms" : 1211617,
    "meter_msg_count_sent" : 0,
    "meter_pkg_count_sent" : 0,
    "time_in_em0_ms" : 516,
    "time_in_em1_ms" : 13,
    "time_in_em2_ms" : 299623
  },
  "hub_attachments" : {
    "meter_pkg_count_recv" : 0,
    "node_version" : "986-49df5891"
  }
}

and a separate metric from the IR reader:
$aws/rules/ingest_tibber_bridge_data/tibber-bridge/yyy/publish/TFD01/dc8e95fffeb82bfe/metric

Code: Select all

{
  "$type" : "node_status",
  "node_status" : {
    "product_id" : 49344,
    "bootloader_version" : 17563650,
    "node_battery_voltage" : 3.458252,
    "node_temperature" : 28.498924,
    "node_avg_rssi" : -47.47422,
    "node_avg_lqi" : 198.903183,
    "acmp_rx_autolevel_300" : 146,
    "acmp_rx_autolevel_9600" : 146,
    "radio_tx_power" : 200,
    "node_uptime_ms" : 1811921,
    "meter_msg_count_sent" : 0,
    "meter_pkg_count_sent" : 0,
    "time_in_em0_ms" : 598,
    "time_in_em1_ms" : 18,
    "time_in_em2_ms" : 299534
  },
  "hub_attachments" : {
    "meter_pkg_count_recv" : 0,
    "node_version" : "986-49df5891"
  }
}

I hacked together a quick python script that forwards all traffic to the actual Tibber MQTT server (and vice versa, but never saw any return traffic):

Code: Select all

import paho.mqtt.client as mqtt

def tibber_to_pulse_message(client, userdata, msg):
    global localclient

    localclient.publish(msg.topic, msg.payload)
    print("Forwarding '%s' to '%s'" % (msg.payload, msg.topic))
    
def pulse_to_tibber_message(localclient, userdata, msg):
    global client
    
    res = client.publish(msg.topic, msg.payload)
        
    if res.rc == mqtt.MQTT_ERR_SUCCESS:
        print("Forwarded '%s' to '%s'" % (msg.payload, msg.topic))
    else:
        print("Publish error %d, reconnecting\n" % res.rc)
        client.reconnect()

def tibber_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected to Tibber MQTT Broker!")
    else:
        print("Failed to connect, return code %d\n", rc)

client = mqtt.Client("3494545e84000")
client.on_message = tibber_to_pulse_message
client.on_connect = tibber_connect
client.tls_set("ca.pem","cert.pem","private.pem")
client.connect("a1zhmn1192zl1a.iot.eu-west-1.amazonaws.com",8883)
client.subscribe("tibber-bridge/yyy/receive/#")

localclient = mqtt.Client("tibberForwarder")
localclient.on_message = pulse_to_tibber_message
localclient.connect("localhost", 1883, 60)
localclient.subscribe("$aws/rules/ingest_tibber_bridge_data/tibber-bridge/#")

while True:
    client.loop(timeout=0.1)
    localclient.loop(timeout=0.1)

As you see I have some reconnection code going on as I get a disconnect after the initial connect. Ét voila: the Pulse icon shows up in the app as soon as I do this.

But the missing piece is still the data. I now read that the eBZ DD3 xxx ODZ1 meter is actually supported and apparently its raw data is just being forwarded to Tibber. Don't know if the latter really holds true and which topic it would be send on. Probably "/data". Might just dump something out and see what happens.

So again, if anyone has a working pulse and configures it to use a local MQTT server I'd be very happy about dumps.

[johu]

Got a bit further with the data. I configured the pulse for a 10000 pulses/kWh input.

Then I flashed an IR LED at it and indeed the MQTT messages now changed:
$aws/rules/ingest_tibber_bridge_data/tibber-bridge/yyy/publish/TFD01/dc8e95fffeb82bfe/imp:

Code: Select all

{"$type": "imp_data", "timestamp_ms": 34652489,"delta_ms": 11782,"kw":0.030555, "kwh": 0.8297}

I hacked up a quick python script that uses the actual meter data to generate an appropriate pulse frequency (the pulse LED on the actual meter is too dim) and let it run all night. Strangely though, the kWh counter does not increase at low wattage, apparently. In the beginning I simulated multiple kW with like 10 Hz frequency that's how the 0.8297 kWh accumulated but then the realistic data didn't increase the counter.

Most of all the tibber app doesn't pick up on it. It says the pulse is not sending data. So could really use data from someone with a working meter.
I have an EBZ DD3 ODZ1 which according to this facebook post works for some people. I flashed its raw data into the pulse with no result.

[johu]

I have finally cracked it. It seems tibber now supports my meter so I was able to intercept some actual data packets.

The data is sent in topic $aws/rules/ingest_tibber_bridge_data/tibber-bridge/xxx/publish/TJH01/yyy/obis_stream
xxx is the long uid and yyy the eui (see metric)

The contents is just the raw data read from the meter with one newline removed

Code: Select all

/EBZ5DD3BZ06ETA_107
1-0:0.0.0*255(1EBZzzzz)
1-0:96.1.0*255(1EBZzzzz)
1-0:1.8.0*255(009094.27916715*kWh)
1-0:16.7.0*255(000169.25*W)
1-0:36.7.0*255(000133.51*W)
1-0:56.7.0*255(000035.74*W)
1-0:76.7.0*255(000000.00*W)
1-0:32.7.0*255(233.4*V)
1-0:52.7.0*255(234.5*V)
1-0:72.7.0*255(234.6*V)
1-0:96.5.0*255(001C0104)
0-0:96.8.0*255(08F0EAB3)
!

I am currently brushing up the code then will publish the python script. You will need to extract the longer tibber uid that is found in the parameter "mqtt_topic", the bridge id that you can obtain by running the "version" command on the console, and the "EUI" of the actual IR device which is found on the "Nodes" tab.
Then you also need to extract the three certificates for the MQTT connection into PEM files.

[johu]

I have published the python script along with some instructions: https://github.com/jsphuebner/esp-egyco ... rol/tibber

[deadrabbit]

Hi Johannes,

i found your hack in the akkudoktor.net forum and your youtube channel.

Because the tibber pulse gets on my nerve (data not confirmend, empty battery, ...), i want to switch to your solution.

Could you give me a hint, how can i get the raw-data of a hichi (aka tasmota smart meter interface) to your script?

As i understand, i need a mqtt topic with the raw data from the meter on my local mqtt server in the config:

Code: Select all

  "meter": { "rawtopic": "/ebz/raw" }

Thanks a lot for your work!

Maximilian

[johu]

Ok cool.
I don't know how to adapt your exact solution to transmit raw data as I hacked up my own with an ESP8266 wifi module: https://github.com/jsphuebner/esp-egyco ... ounter.ino

Hope that somehow helps

[deadrabbit]

Yes, that would help. I can also use your code for my esp8266.

Anoter problem is, i'm getting error from the python script:

Code: Select all

Traceback (most recent call last):
  File "/home/kerberos/tibber/tibbersend.py", line 72, in <module>
    client.tls_set("tibber/ca.pem", "tibber/cert.pem", "tibber/private.pem")
  File "/usr/lib/python3/dist-packages/paho/mqtt/client.py", line 823, in tls_set
    context.load_cert_chain(certfile, keyfile)
ssl.SSLError: [SSL] PEM lib (_ssl.c:3921)

It looks like there is some trouble with the cert file. I've formated them in a way like:

Code: Select all

-----BEGIN CERTIFICATE-----
MIIDWTdCCAkGgAwIBAgIdUIJOZe34TREvLMBOZ5MYK6KSdFyd0wDQYJKoZIhvcNAQELBQAwTTFLMEkGA1UECwxCQW1hem9uIFdlYiBTZXJ2aWNlcyBPPUFtYXpvbi
..................

MCwo9hDUndEbG9vapXID4Hg37sdhuyZfUbda9D
-----END CERTIFICATE-----

Is this correct?

[johu]

Yes I also have a line break after BEGIN --- and before --- END
All else is in one line

[deadrabbit]

Okay, i think i made some copy and paste error.

But the main problem still exists: How can i get the raw data to mqtt topic? I've tried to compile your esp8266 firmware from here:
https://github.com/jsphuebner/esp-egyco ... ounter.ino

But i could not find any libary in Arduino IDE which is called

Code: Select all

StringReadStream.h

. Where did you get it from? Did you use the arduino IDE?

[johu]

Yes I used just Arduino IDE. The lib is called StreamLib

[deadrabbit]

Hi Johannes,

i'm still struggling getting the SML data to the mqtt topic. ATM i've attached a UART IR reading head to my electricity meter.

In NodeRed i'm able to read data:

But I've no idea how i should convert to the data for pubishing it to an mqtt-topic.

Could you give me a hint?

[johu]

Is your meter not producing ASCII text output?

If so maybe this helps: https://github.com/jsphuebner/esp-egycounter/issues/1